You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "geserdugarov (via GitHub)" <gi...@apache.org> on 2025/01/23 12:55:29 UTC

[PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

geserdugarov opened a new pull request, #12697:
URL: https://github.com/apache/hudi/pull/12697

   ### Change Logs
   
   Proposed optimization to reduce SerDe costs for Flink operators.
   
   Proof of concept was presented in the corresponding [claim of RFC](https://github.com/apache/hudi/pull/12550).
   For stream write into Hudi with simple bucket index, total write time decreased by 15%, which is significant for stream processing.
   
   ### Impact
   
   None for this stage.
   
   ### Risk level (write none, low medium or high below)
   
   None for this stage.
   
   ### Documentation Update
   
   No need for this stage.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Change Logs and Impact were stated clearly
   - [x] Adequate tests were added if applicable
   - [x] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971334857


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   @cshuo , at this place you suggested to pass `StringData` and add corresponding constructor. My quick check showed performance decrease for this case, but we agreed to postpone research of the cause.
   
   I've done this research, and found that the reason we shouldn't implement this optimization is garbage collection. I suppose, when we postpone creation of Java `String` objects in writers, it leads to creation of huge number of objects when our buffers are full, and at this point we also need to create a lot of `String`s. It leads to high momentum consumption of memory, and causes more full GC run, which stop-the-world, and decreases total performance.
    
   cc @danny0405 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2633260693

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 845b519c5d54990eca290c5ecb014fed26e76eaf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319) 
   * 5e0f4ee7a09c776907277cac883c60dcd00384d9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1942163691


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @cshuo , hi! If doesn't bother you, could you, please, check this RFC for design conflicts with your RFC, which is in progress. If there is no conflicts, I propose to move these changes further.
   Otherwise, could you, please, provide some drafts, to start to work in collaboration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2649768177

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 53d5287b764eb797f8598539dccdebea4fd2a856 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427) 
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971334857


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   @cshuo , at this place you suggested to pass `StringData` and add corresponding constructor. My quick check showed performance decrease for this case, but I postponed research of it.
   
   I've done this research, and found that the reason we shouldn't implement this optimization is garbage collection. I suppose, when we postpone creation of Java `String` objects in writers, it leads to creation of huge number of objects when our buffers are full, and at this point we also need to create a lot of `String`s. It leads to high momentum consumption of memory, and causes more full GC run, which stop-the-world, and decreases total performance.
    
   cc @danny0405 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1970722231


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,328 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @wombatu-kun
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.

Review Comment:
   Yes, I will update this today. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2619483667

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * cae222650ba438e160f735078734f4166b9e93f0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2671393383

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740",
       "triggerID" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701) 
   * d25ddf940c1c4ce72c18be440591fd8b432d236d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2627613905

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * af1ce0ffef7e766e8d78e6056d907a46bff75f49 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241) 
   * 845b519c5d54990eca290c5ecb014fed26e76eaf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2619736718

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * cae222650ba438e160f735078734f4166b9e93f0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2667735720

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2640214275

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 53d5287b764eb797f8598539dccdebea4fd2a856 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2654317851

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f1218bb1bb1541f9d59b932100883f1c9a4936e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12697:
URL: https://github.com/apache/hudi/pull/12697


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1932042819


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @danny0405 , PR is ready: https://github.com/apache/hudi/pull/12722.
   In the description to this PR, I've described costs from this extra conversion:
   > These costs could be accepted for now due to acceptable values: 5 798 CPU samples from 183 236 in total, which is about 3%.
   
   Total performance improvement:
   - total write time decreased from 344 s to 265 s, which is about 23%,
   - data passed between Flink operators decreased from 19.4 GB to 12.9 GB, which is about 33.5%.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1929942302


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   Is there already a POC now, can we do a micro-benchmark to prove the gains.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2627820822

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 845b519c5d54990eca290c5ecb014fed26e76eaf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2650022594

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2652545625

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 10e121823db9aa4d0a117909d29798c9ac502f80 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1944654396


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @geserdugarov here is RFC pr #12795, welcome to review and get involved to collaborate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2653991786

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e121823db9aa4d0a117909d29798c9ac502f80 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557) 
   * 5f1218bb1bb1541f9d59b932100883f1c9a4936e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2650023588

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1957519782


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   Is it possible that `HoodieFlinkInternalRow` implements Flink internal `RowData`, and during constructing streaming pipelines, we can use `InternalTypeInfo` to represent the output transforming type. Two benefits come to my mind:
   * For sake of performance, `InternalTypeInfo` with `RowDataSerializer` is no doubt the most efficient one.
   * No need to introduce custom TypeInfo or serializer, to minimize the risk of bugs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1959009278


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   @geserdugarov Got it. So the main concern is the `serde issues with DataStream::keyBy`, could you please provide more details about it, or provide the related test case to reproduce it? I'm willing to take a look at it, and maybe it can be solved some way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1959009278


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   @geserdugarov Got it. So the main concern is the `serde issues with DataStream::keyBy`, could you please provide more details about it, or provide the related test case to reproduce it? I'm willing to take a look at it, and maybe it can be solved some way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971334857


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   @cshuo , at this place you suggested to pass `StringData` and add corresponding constructor. My quick check showed performance decrease for this case, but we agreed to postpone research of the cause.
   
   I've done this research, and found that the reason we shouldn't implement this optimization is related to garbage collection. I suppose, when we postpone creation of Java `String` objects in writers, it leads to creation of huge number of objects when our buffers are full, and at this point we also need to create a lot of `String`s. It leads to high momentum consumption of memory, and causes more full GC run, which decreases total performance.
    
   cc @danny0405 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2609744288

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e375fd515583dffce316cdffe2c923ab3a86e796 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2614760866

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e375fd515583dffce316cdffe2c923ab3a86e796 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046) 
   * cae222650ba438e160f735078734f4166b9e93f0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2614759809

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e375fd515583dffce316cdffe2c923ab3a86e796 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046) 
   * cae222650ba438e160f735078734f4166b9e93f0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2620712913

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * af1ce0ffef7e766e8d78e6056d907a46bff75f49 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1960099516


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   Unfortunately, I don't see corresponding commit here. Looks like git cleaner runs often here. And in my local history it's a tough task to find it after all rebases, squashes, and resets. I didn't save it due to the fact that it was a dead end for me. And I've started to rewrite everything.
   
   To reproduce this issue, I suppose, it should be enough to construct custom pipeline from Flink operators. Convert in the first one to some data structure for testing, add `keyBy` shuffling using some data from new data structure, and then write data stream somewhere, or, maybe, black hole could be used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971337841


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   ![Heap analysis - comment](https://github.com/user-attachments/assets/a59b5872-3562-4d87-ba48-44aa26e43584)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2649766586

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 53d5287b764eb797f8598539dccdebea4fd2a856 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427) 
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2649871852

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2651404662

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541) 
   * 10e121823db9aa4d0a117909d29798c9ac502f80 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2610036096

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e375fd515583dffce316cdffe2c923ab3a86e796 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2683896237

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740",
       "triggerID" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a353d0721220d2bb09b0d0272291ca9035ecd986",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3907",
       "triggerID" : "a353d0721220d2bb09b0d0272291ca9035ecd986",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a353d0721220d2bb09b0d0272291ca9035ecd986 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3907) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1934943752


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @cshuo is recently working on a new RFC to add basic abstractions of schema/data type/expressions to Hudi, so that we can integrate with the engine specific "row" for both the writer and reader, the design doc would be coming out, will cc you if you have intreast in it, it's a huge task and maybe you can help with it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2627617245

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * af1ce0ffef7e766e8d78e6056d907a46bff75f49 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241) 
   * 845b519c5d54990eca290c5ecb014fed26e76eaf Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2639932416

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5e0f4ee7a09c776907277cac883c60dcd00384d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372) 
   * 53d5287b764eb797f8598539dccdebea4fd2a856 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2650268763

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2667463467

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f1218bb1bb1541f9d59b932100883f1c9a4936e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565) 
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2671618338

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740",
       "triggerID" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d25ddf940c1c4ce72c18be440591fd8b432d236d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2652652255

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 10e121823db9aa4d0a117909d29798c9ac502f80 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2651414298

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c7a7542325e539df7f9f5ff373f4673e8fb1f594 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541) 
   * 10e121823db9aa4d0a117909d29798c9ac502f80 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2614807126

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cae222650ba438e160f735078734f4166b9e93f0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1929943731


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   Yes, but I will work on it a little bit more, and will provide benchmark results with profiling on this week. Also I will provide corresponding PR to check code changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1961659048


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   @geserdugarov got it. I'm also good with customized typeInfo or serializer in terms of performance, anyway, let's move forward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2667962486

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971337841


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   ![Heap analysis - comment](https://github.com/user-attachments/assets/d7b25df1-4dae-4e69-b7c4-1dca3d657a6f)
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2633264500

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 845b519c5d54990eca290c5ecb014fed26e76eaf Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319) 
   * 5e0f4ee7a09c776907277cac883c60dcd00384d9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2633536650

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5e0f4ee7a09c776907277cac883c60dcd00384d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1942163691


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @cshuo , hi! If doesn't bother you, could you, please, check this RFC for design conflicts with your RFC, which is in progress. If there is no conflicts, I propose to move these changes further.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1942200478


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @geserdugarov Sorry for the late reply, just came back from the lunar new year holiday :) I'll take a look at your rfc and benchmark code soon. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971334857


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   @cshuo , here you suggested to pass `StringData` and add corresponding constructor. My quick check showed performance decrease for this case, but I postponed research of it.
   
   I've done this research, and found that the reason we should do it is garbage collection. When we postpone creation of Java `String` objects in writers, it leads to move of huge number of objects (related to `StringData`) from young generation to old generation. Consequently, it causes more full GC runs, and decreases performance.
    
   cc @danny0405 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971334857


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   @cshuo , here you suggested to pass `StringData` and add corresponding constructor. My quick check showed performance decrease for this case, but I postponed research of it.
   
   I've done this research, and found that the reason we should do it is garbage collection. When we postpone creation of Java `String` objects in writers, it leads to move of huge number of objects (related to `StringData`) from young generation to old generation. Consequently, it causes more full GC runs, and decreases performance.
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1932042819


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,113 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into `Tuple(Tuple(StringData, ..., StringData), RowData)` 
+instead of `HoodieRecord`, where the first sub-tuple will contain necessary metadata. 
+In this case return type could be described as `TupleTypeInfo<>(TupleTypeInfo<>(...), dataStream.getType()))`, which will force Flink to use internal `TupleSerializer`.
+
+### Potential problems
+
+1. Key generators are hardly coupled with Avro `GenericRecord`. 
+   Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key.

Review Comment:
   @danny0405 , PR is ready: https://github.com/apache/hudi/pull/12722.
   In the description to this PR, I've mentioned costs from this extra conversion:
   > These costs could be accepted for now due to acceptable values: 5 798 CPU samples from 183 236 in total, which is about 3%.
   
   Total performance improvement:
   - total write time decreased from 344 s to 265 s, which is about 23%,
   - data passed between Flink operators decreased from 19.4 GB to 12.9 GB, which is about 33.5%.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2620521899

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cae222650ba438e160f735078734f4166b9e93f0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220) 
   * af1ce0ffef7e766e8d78e6056d907a46bff75f49 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2620523137

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * cae222650ba438e160f735078734f4166b9e93f0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220) 
   * af1ce0ffef7e766e8d78e6056d907a46bff75f49 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2639936994

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5e0f4ee7a09c776907277cac883c60dcd00384d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372) 
   * 53d5287b764eb797f8598539dccdebea4fd2a856 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2652544686

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2651661016

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e121823db9aa4d0a117909d29798c9ac502f80 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1960099516


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   Unfortunately, I don't see corresponding commit here. Looks like git cleaner runs often in GitHub. And in my local history it's a tough task to find it after all rebases, squashes, and resets. I didn't save it due to the fact that it was a dead end for me. And I've started to rewrite everything.
   
   To check it, I suppose, it could be enough to add `keyBy` shuffling after conversion into some implementation of `AbstractHoodieRowData`, and add some Flink operator further down.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2671389690

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701) 
   * d25ddf940c1c4ce72c18be440591fd8b432d236d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2667734651

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2609747543

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e375fd515583dffce316cdffe2c923ab3a86e796 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2683797940

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740",
       "triggerID" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a353d0721220d2bb09b0d0272291ca9035ecd986",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3907",
       "triggerID" : "a353d0721220d2bb09b0d0272291ca9035ecd986",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d25ddf940c1c4ce72c18be440591fd8b432d236d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740) 
   * a353d0721220d2bb09b0d0272291ca9035ecd986 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3907) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2619480303

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1959012593


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   @geserdugarov Got it. But there is some differences, what I mean is implementing `HoodieFlinkInternalRow` as a Flink `RowData` like current `AbstractHoodieRowData` in Hudi, not a `Tuple`.  Anyway, could you please provide more details about the serde issue, or provide the related test case to reproduce it? I'm willing to take a look at it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1958998694


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   Actually, I though the same initially, I didn't want to implement any custom type info or serializer. But I chose `HoodieFlinkInternalRow`, which implements Flink internal `Tuple` with nested `RowData`. And it worked great until I faced serde issues with `DataStream::keyBy`. At that time, I didn't found any workarounds how to solve arisen issues during `KeyedProcessOperator::setKeyContextElement` in Flink without implementation of custom serializer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1957519782


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,335 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @xiarixiaoyao
+- @yuzhaojing
+- @wombatu-kun
 
-## Status: Claim
+## Status
+
+Design is under discussion.
+
+Implementation is ready for non bucket and simple bucket, and wait for review:
+https://github.com/apache/hudi/pull/12796
 
 JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+There are seven different categories of 
+[supported data types](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/types_serialization/) 
+in Flink: tuples, POJOs, primitive types, regular classes, values, Hadoop writables, and special types. 
+Among presented data types, tuples are less flexible, but offer the best performance. 
+In this case there is no need in custom type description and custom serializer, and we could use already presented in Flink 
+[`TupleTypeInfo`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/TupleTypeInfo.java) 
+and 
+[`TupleSerializer`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/TupleSerializer.java).
+(All links to Flink documentation or code are provided for version 1.20.) 
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`, so our main focus should be on the upper part of the presented schema.
+
+Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations.
+For chained operator there is no serialization/deserialization between combined transformations.
+But transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't be chained.
+Those partitioners are marked as purple blocks in the schema, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Necessary for processing Hudi metadata is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion because this class doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:

Review Comment:
   Is it possible that `HoodieFlinkInternalRow` implements Flink internal `RowData`, and during constructing streaming pipelines, we can use the native`InternalTypeInfo` to represent the output transforming type. Two benefits come to my mind:
   * For sake of performance, `InternalTypeInfo` with `RowDataSerializer` is no doubt the most efficient one.
   * No need to introduce custom TypeInfo or serializer, to minimize the risk of bugs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2683796375

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3701",
       "triggerID" : "2667734651",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740",
       "triggerID" : "d25ddf940c1c4ce72c18be440591fd8b432d236d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a353d0721220d2bb09b0d0272291ca9035ecd986",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a353d0721220d2bb09b0d0272291ca9035ecd986",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d25ddf940c1c4ce72c18be440591fd8b432d236d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3740) 
   * a353d0721220d2bb09b0d0272291ca9035ecd986 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1970841537


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,328 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @wombatu-kun
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.

Review Comment:
   @danny0405 , I've updated everything here by a353d0721220d2bb09b0d0272291ca9035ecd986.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1971334857


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,344 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @cshuo
 
-## Status: Claim
+## Status
 
-JIRA: [HUDI-8799](https://issues.apache.org/jira/browse/HUDI-8799)
+Main part of implementation is done:
+https://github.com/apache/hudi/pull/12796
+
+Umbrella ticket: [HUDI-8920](https://issues.apache.org/jira/browse/HUDI-8920)
+
+## Abstract
+
+Currently, in the majority of scenarios when Flink writes into Hudi, the first step is row data conversion into Avro record, which is used for key generation, 
+and passed to the following operators as part of `HoodieRecord`. Kryo serializer is used to serialize/deserialize those records. 
+And as it mentioned in the [claim for this RFC](https://github.com/apache/hudi/pull/12550), Kryo serde costs are huge, which is unacceptable for stream processing.
+
+This RFC suggests to implement data processing with keeping focus on performance for Flink, and considering Flink's internal data types and serialization.
+
+## Background
+
+Currently, `HoodieRecord` is chosen as standardized API for interacting with a single record, see [RFC-46](../rfc-46/rfc-46.md). 
+But `HoodieRecord` complex structure leads to high serialization/deserialization costs if it needed to be sent.
+So for Flink main use case scenario of stream processing, when we handle each record separately on different operators, 
+current approach with initial conversion into `HoodieRecord` becomes unacceptable.
+
+Conversion into `HoodieRecord` should be done only in operators, which actually perform write into HDFS, S3, etc., to prevent excessive costs. 
+And also it allows to implement future optimizations with direct write of Flink `RowData` to parquet files if it is needed without any intermediate conversions.
+In Flink pipelines we could keep internal `RowData` together with only necessary Hudi metadata.
+And these metadata should be added considering Flink data types.
+
+## Implementation
+
+To prepare implementation plan we should start from current state review.
+
+### Write
+
+All scenarios of Flink writes into Hudi could be presented in a schema below:
+
+![`DataStream` for `HoodieTableSink`](datastream_hoodietablesink.png)
+
+There are two special cases for processing: bulk insert, and append mode (insert operation into MOR and COW without inline clustering), which are seen in the lower part.
+For both of these cases there is no conversion of `RowData` into `HoodieRecord`.
+Also, Flink could automatically chain operators together if it's possible, which means that one operator combines multiple transformations,
+and there are no serialization/deserialization between these transformations.
+Therefore, our main focus should be on the upper part of the presented schema due to the fact that for bulk insert and append mode chained Flink operators are used.
+
+Transformations separated by some partitioners, like 
+[`keyBy()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L304-L308), 
+which uses 
+[`KeyGroupStreamPartitioner`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/KeyGroupStreamPartitioner.java), 
+or 
+[`partitionCustom()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L398-L402), 
+which expects user defined custom partitioner, couldn't been chained.
+Those partitioners are marked as purple blocks in the schema above, and at those places we will face high serialization/deserialization costs.
+
+### Read
+
+As for reading of Hudi table by Flink, at the first blush there is no need to implement such optimizations there.
+
+### Metadata
+
+We start writing into Hudi table from `DataStream<RowData>`.
+Hudi metadata, which is necessary for processing in different operators, is marked by red color on the schema above.
+We could use 
+[`map()`](https://github.com/apache/flink/blob/b1fe7b4099497f02b4658df7c3de8e45b62b7e21/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java#L611-L614)
+transformation to convert incoming `RowData` into a new object `HoodieFlinkInternalRow`.
+We don't use `HoodieFlinkRecord` name to prevent confusion, because `HoodieFlinkInternalRow` doesn't extend `HoodieRecord`.
+
+Structure of `HoodieFlinkInternalRow`:
+```Java
+public class HoodieFlinkInternalRow implements Serializable {
+
+  private static final long serialVersionUID = 1L;
+
+  // the number of fields without nesting
+  protected static final int ARITY = 7;
+
+  // recordKey, partitionPath, isIndexRecord and rowData are final
+  private final StringData recordKey;
+  private final StringData partitionPath;
+  private StringData fileId;
+  private StringData instantTime;
+  private StringData operationType;
+  // there is no rowData for index record
+  private final BooleanValue isIndexRecord;
+  private final RowData rowData;
+
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, RowData rowData) {
+    this(recordKey, partitionPath, "", "", "", false, rowData);
+  }
+
+  // constructor for index records without row data
+  public HoodieFlinkInternalRow(String recordKey, String partitionPath, String fileId, String instantTime) {
+    this(recordKey, partitionPath, fileId, instantTime, "", true, null);
+  }
+
+  public HoodieFlinkInternalRow(String recordKey,
+                                String partitionPath,
+                                String fileId,
+                                String instantTime,
+                                String operationType,
+                                boolean isIndexRecord,
+                                RowData rowData) {
+    this.recordKey = StringData.fromString(recordKey);
+    this.partitionPath = StringData.fromString(partitionPath);
+    this.fileId = StringData.fromString(fileId);
+    this.instantTime = StringData.fromString(instantTime);
+    this.operationType = StringData.fromString(operationType);
+    this.isIndexRecord = new BooleanValue(isIndexRecord);
+    this.rowData = rowData;
+  }
+
+  public String getRecordKey() {}
+
+  public String getPartitionPath() {}
+
+  public void setFileId(String fileId) {}
+  public String getFileId() {}
+
+  public void setInstantTime(String instantTime) {}
+  public String getInstantTime() {}
+
+  public void setOperationType(String operationType) {}
+  public String getOperationType() {}
+
+  public boolean isIndexRecord() {}
+
+  public RowData getRowData() {}
+
+  public HoodieFlinkInternalRow copy(RowDataSerializer rowDataSerializer) {
+    return new HoodieFlinkInternalRow(
+        this.recordKey.toString(),
+        this.partitionPath.toString(),
+        this.fileId.toString(),
+        this.instantTime.toString(),
+        this.operationType.toString(),
+        this.isIndexRecord.getValue(),
+        rowDataSerializer.copy(this.rowData));
+  }
+}
+```
+
+To describe how to serialize and deserialize it properly, we also need to implement `HoodieFlinkInternalRowTypeInfo` and `HoodieFlinkInternalRowSerializer`.
+
+```Java
+public class HoodieFlinkInternalRowTypeInfo extends TypeInformation<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  private final RowType rowType;
+
+  public HoodieFlinkInternalRowTypeInfo(RowType rowType) {
+    this.rowType = rowType;
+  }
+
+  @Override
+  public boolean isBasicType() {}
+
+  @Override
+  public boolean isTupleType() {}
+
+  @Override
+  public int getArity() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  /**
+   * Used only in Flink `CompositeType`, not used in this type
+   */
+  @Override
+  public int getTotalFields() {
+    return HoodieFlinkInternalRow.ARITY;
+  }
+
+  @Override
+  public Class<HoodieFlinkInternalRow> getTypeClass() {}
+
+  @Override
+  public boolean isKeyType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> createSerializer(ExecutionConfig config) {
+    return new HoodieFlinkInternalRowSerializer(this.rowType);
+  }
+
+  @Override
+  public String toString() {}
+
+  @Override
+  public boolean equals(Object obj) {}
+
+  @Override
+  public int hashCode() {}
+
+  @Override
+  public boolean canEqual(Object obj) {}
+}
+```
+
+```Java
+public class HoodieFlinkInternalRowSerializer extends TypeSerializer<HoodieFlinkInternalRow> {
+
+  private static final long serialVersionUID = 1L;
+
+  protected RowType rowType;
+
+  protected RowDataSerializer rowDataSerializer;
+
+  protected StringDataSerializer stringDataSerializer;
+
+  public HoodieFlinkInternalRowSerializer(RowType rowType) {
+    this.rowType = rowType;
+    this.rowDataSerializer = new RowDataSerializer(rowType);
+    this.stringDataSerializer = StringDataSerializer.INSTANCE;
+  }
+
+  @Override
+  public boolean isImmutableType() {}
+
+  @Override
+  public TypeSerializer<HoodieFlinkInternalRow> duplicate() {}
+
+  @Override
+  public HoodieFlinkInternalRow createInstance() {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow creation with some defaults.");
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from) {
+    return from.copy(rowDataSerializer);
+  }
+
+  @Override
+  public HoodieFlinkInternalRow copy(HoodieFlinkInternalRow from, HoodieFlinkInternalRow reuse) {
+    throw new UnsupportedOperationException("HoodieFlinkInternalRow doesn't allow reusing.");
+  }
+
+  @Override
+  public int getLength() {
+    // -1 for variable length data types
+    return -1;
+  }
+
+  @Override
+  public void serialize(HoodieFlinkInternalRow record, DataOutputView target) throws IOException {
+    boolean isIndexRecord = record.isIndexRecord();
+    target.writeBoolean(isIndexRecord);
+    stringDataSerializer.serialize(StringData.fromString(record.getRecordKey()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getPartitionPath()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getFileId()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getInstantTime()), target);
+    stringDataSerializer.serialize(StringData.fromString(record.getOperationType()), target);
+    if (!isIndexRecord) {
+      rowDataSerializer.serialize(record.getRowData(), target);
+    }
+  }
+
+  @Override
+  public HoodieFlinkInternalRow deserialize(DataInputView source) throws IOException {
+    boolean isIndexRecord = source.readBoolean();
+    StringData recordKey = stringDataSerializer.deserialize(source);
+    StringData partition = stringDataSerializer.deserialize(source);
+    StringData fileId = stringDataSerializer.deserialize(source);
+    StringData instantTime = stringDataSerializer.deserialize(source);
+    StringData operationType = stringDataSerializer.deserialize(source);
+    HoodieFlinkInternalRow record;
+    if (!isIndexRecord) {
+      RowData rowData = rowDataSerializer.deserialize(source);
+      record = new HoodieFlinkInternalRow(
+          recordKey.toString(),

Review Comment:
   @cshuo , at this place you suggested to pass `StringData` and add corresponding constructor. My quick check showed performance decrease for this case, but we agreed to postpone research of the cause.
   
   I've done this research, and found that the reason we shouldn't implement this optimization is garbage collection. I suppose, when we postpone creation of Java `String` objects in writers, it leads to creation of huge number of objects when our buffers are full, and at this point we also need to create a lot of `String`s. It leads to high momentum consumption of memory, and causes more full GC run, which decreases total performance.
    
   cc @danny0405 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "geserdugarov (via GitHub)" <gi...@apache.org>.
geserdugarov commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2639950653

   @danny0405 , @cshuo , I've updated description here with added structure of `HoodieFlinkRecordTypeInfo ` and `HoodieFlinkRecordSerializer`.
   I've already implemented optimization for simple bucket index and non bucket case. Opened corresponding PR: https://github.com/apache/hudi/pull/12796.
   First version, that I could do everything without custom serializer was wrong. I've faced issues with serde during conversion of `DataStream` into `KeyedStream`. With implemented `HoodieFlinkRecordTypeInfo ` and `HoodieFlinkRecordSerializer` everything works correctly. And for non bucket case I got 31% performance improvement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2653985146

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 10e121823db9aa4d0a117909d29798c9ac502f80 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557) 
   * 5f1218bb1bb1541f9d59b932100883f1c9a4936e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2667461804

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5f1218bb1bb1541f9d59b932100883f1c9a4936e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565) 
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12697:
URL: https://github.com/apache/hudi/pull/12697#issuecomment-2667573988

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3046",
       "triggerID" : "e375fd515583dffce316cdffe2c923ab3a86e796",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3151",
       "triggerID" : "cae222650ba438e160f735078734f4166b9e93f0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cae222650ba438e160f735078734f4166b9e93f0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3220",
       "triggerID" : "2619480303",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3241",
       "triggerID" : "af1ce0ffef7e766e8d78e6056d907a46bff75f49",
       "triggerType" : "PUSH"
     }, {
       "hash" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3319",
       "triggerID" : "845b519c5d54990eca290c5ecb014fed26e76eaf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3372",
       "triggerID" : "5e0f4ee7a09c776907277cac883c60dcd00384d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3427",
       "triggerID" : "53d5287b764eb797f8598539dccdebea4fd2a856",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3534",
       "triggerID" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c7a7542325e539df7f9f5ff373f4673e8fb1f594",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3541",
       "triggerID" : "2650022594",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3547",
       "triggerID" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "triggerType" : "PUSH"
     }, {
       "hash" : "10e121823db9aa4d0a117909d29798c9ac502f80",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3557",
       "triggerID" : "2652544686",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3565",
       "triggerID" : "5f1218bb1bb1541f9d59b932100883f1c9a4936e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695",
       "triggerID" : "8fcebf765ef90f1ea6e2d025deda423ae8e29655",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8fcebf765ef90f1ea6e2d025deda423ae8e29655 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=3695) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-8799] Design of RFC-84, Optimized SerDe of `DataStream` in Flink operators [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12697:
URL: https://github.com/apache/hudi/pull/12697#discussion_r1970714544


##########
rfc/rfc-84/rfc-84.md:
##########
@@ -24,8 +24,328 @@
 ## Approvers
 
 - @danny0405
-- @...
+- @wombatu-kun
+- @yuzhaojing
 
-## Status: Claim
+## Status
+
+Design is under discussion.

Review Comment:
   This is out-of-date ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org