You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2026/02/04 21:43:12 UTC
[PR] feat(flink): Implement continuous sorting feature for append write [hudi]
prashantwason opened a new pull request, #18083:
URL: https://github.com/apache/hudi/pull/18083
### Describe the issue this Pull Request addresses
This PR implements a continuous sorting feature for the Flink append write sink. The continuous sorting mode maintains sorted order incrementally using a TreeMap, which avoids large pause times from batch sorting and reduces single-partition lag during ingestion by minimizing backpressure.
### Summary and Changelog
**Summary:** Added a continuous sorting mode (`AppendWriteFunctionWithContinuousSort`) that provides non-blocking O(log n) inserts and incremental draining, offering predictable latency without sort spikes.
**Changelog:**
- Added `AppendWriteFunctionWithContinuousSort` class that keeps records in a TreeMap keyed by a code-generated normalized key and an insertion sequence
- When buffer reaches max capacity, oldest entries are drained and written immediately
- Updated `AppendWriteFunctions.create` to instantiate the continuous sorter when `WRITE_BUFFER_SORT_CONTINUOUS_ENABLED` is true
- Introduced new FlinkOptions:
- `write.buffer.sort.continuous.enabled` - Whether to use continuous sorting instead of batch sorting
- `write.buffer.sort.continuous.drain.size` - Number of records to drain each time max capacity is reached
- Added `ITTestAppendWriteFunctionWithContinuousSort` integration tests covering buffer flush triggers, sorted output correctness, drain behaviors, and invalid-parameter error cases
### Impact
**New Configuration Options:**
- `write.buffer.sort.continuous.enabled` (default: `false`) - Enables continuous sorting mode
- `write.buffer.sort.continuous.drain.size` (default: `1`) - Controls drain batch size
No breaking changes to existing functionality. The feature is disabled by default.
### Risk Level
low - This is a new optional feature that is disabled by default. Existing behavior is unchanged unless the user explicitly enables continuous sorting.
### Documentation Update
The config description is included in the code. Documentation update for the Hudi website may be needed to describe the new continuous sorting feature and its configuration options.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4084727096
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cdd7c6dfbb34164cd915eb89f746db7a496381a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529)
* a4085261407b9451e19565f5f1001f251dc0f7c7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3949598053
Thanks @cshuo for the review!
**Regarding async write support:**
You're correct that this PR doesn't include async write like #13892. The goal of this PR is more focused - it specifically addresses the sorting bottleneck in append write by switching from batch sorting to continuous (incremental) sorting. This eliminates the large pause times that occur when sorting a full buffer.
The key benefits of continuous sorting:
- **Non-blocking O(log n) inserts** vs O(n log n) batch sort at flush time
- **Predictable latency** - no sort spikes when buffer fills up
- **Incremental draining** - oldest entries are drained and written immediately when buffer reaches max capacity
- **Reduced backpressure** - minimizes single-partition lag during ingestion
**Regarding benchmarks:**
I don't have formal benchmark numbers comparing to the existing batch sort approach. The primary motivation was to eliminate the unpredictable latency spikes from batch sorting rather than overall throughput improvement. That said, adding async write on top of continuous sorting would be a natural extension that could improve throughput further.
Would you like me to add some basic benchmarking to quantify the latency improvements? Or do you have specific concerns about the approach that I should address first?
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4174050645
> hey @cshuo: can you re-review the patch.
@nsivabalan Except for the last two comments, which are not resolved yet @prashantwason , the PR LGTM overall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2985775692
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ int cmp = normalizedKeyComputer.compareKey(k1.keySegment, 0, k2.keySegment, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().getExecutionConfig().isObjectReuseEnabled();
Review Comment:
There is some compile error for higher flink versions.
```
Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure
/home/runner/work/hudi/hudi/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:[158,50] cannot find symbol
symbol: method getExecutionConfig()
```
You can refer to `RuntimeContextUtils` to adapt object reuse check for different flink versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3857495189
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3949854378
@prashantwason thks for the detail explanation.
> Would you like me to add some basic benchmarking to quantify the latency improvements? Or do you have specific concerns about the approach that I should address first?
Regarding the append write with buffer sorting, there are already two approaches, `AppendWriteFunctionWithBIMBufferSort` and `AppendWriteFunctionWithDisruptorBufferSort` in the repository. Both implement sorting and flushing in an asynchronous manner, aiming to mitigate spike issues during the flush process.
This PR adopts a continuous sorting approach, which can indeed alleviate sorting spikes. But the flush process involves not only sorting but also file-writing overhead and continuous sort will spread sorting overhead across each record writing process. Therefore, Iād like to understand how much performance gain this PR will bring compared to the existed functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3949602508
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2955401630
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
Review Comment:
Done. Made `reusableKeyBuffer` a local variable in `open()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4030999358
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3cdd7c6dfbb34164cd915eb89f746db7a496381a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2955400488
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
Review Comment:
Done. Now using `AppendWriteFunctions.resolveSortKeys(config)` which falls back to record key when `write.buffer.sort.keys` is not set.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
+ if (sortKeys == null || sortKeys.trim().isEmpty()) {
+ throw new IllegalArgumentException("Sort keys cannot be null or empty for continuous sort");
+ }
+
+ List<String> sortKeyList = Arrays.stream(sortKeys.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+
+ if (sortKeyList.isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Sort keys list is empty after parsing: '%s'", sortKeys));
+ }
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ this.reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
Review Comment:
Done. Added object reuse detection via `getRuntimeContext().isObjectReuseEnabled()` and `RowDataSerializer.copy()` in `processElement()` when object reuse is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3857574020
@prashantwason thks for the contribution.
A high-level question before reviewing the details, I noticed this PR doesn't support async write like #13892, and actually the time cost of flushing records to file will be greater than sorting. Do you have any benchmark number about perf comparison of this PR and the previous version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4123329639
@cshuo All review comments from the latest round have been addressed in commit e5921668:
1. **`super.snapshotState()` / `super.endInput()` out of `finally`**: Moved both out of `finally` clauses to match other append write functions ā write meta event is no longer sent to coordinator when there is an ingestion exception.
2. **`isObjectReuseEnabled()` compilation on Flink 1.17/1.18**: Changed to `getRuntimeContext().getExecutionConfig().isObjectReuseEnabled()` for compatibility with lower Flink versions.
3. **MemorySegment allocation in TreeMap comparator**: Pre-wrapped `MemorySegment` is now stored in `SortKey.keySegment` to avoid per-comparison allocation and reduce GC pressure.
4. **Test with object reuse enabled**: Added `testObjectReuseEnabled` test that verifies records are correctly copied when Flink object reuse is enabled (prevents TreeMap corruption from mutated references).
Please take another look when you get a chance. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3036438769
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
Review Comment:
š¤ nit: The check `if (cmp != 0)` is a common pattern, but it's redundant. `recordComparator.compare` and `Long.compare` also return 0 if equal, so you could directly return their results if `cmp == 0` without an explicit check.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
Review Comment:
š¤ nit: It might be clearer to initialize `insertionSequence` to `0L` when `sortedRecords` is cleared in `snapshotState()` and `endInput()` to consistently reflect a fresh state, similar to how it's done in the constructor. Or, is it intended to continue the sequence?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
Review Comment:
š¤ nit: Could `maxCapacity` and `drainSize` be made `final` as they are configured once and not changed?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctions.java:
##########
Review Comment:
**Line 97:** š¤ nit: The `s -> !s.isEmpty()` filter is good. Could we also add a `.trim()` to this stream pipeline before filtering, to handle cases like `"key1, ,key2"` that would result in an empty string after trimming?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2972558964
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
Review Comment:
Let's move `super.snapshotState();` out of the `finally` clause like other append write functions, since write meta event should not be sent to coordinator when there is ingestion exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4124946211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
}, {
"hash" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5921668dfe6aeba6ae085dfd9ae79371e144ae4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796)
* e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2986893643
##########
hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/utils/RuntimeContextUtils.java:
##########
@@ -55,4 +56,10 @@ public static long getWatermarkInternal(RuntimeContext runtimeContext) {
return Long.parseLong(jobParameters.getOrDefault(AUTO_WATERMARK_INTERVAL.key(),
AUTO_WATERMARK_INTERVAL.defaultValue().toMillis() + ""));
}
+
+ public static boolean isObjectReuseEnabled(RuntimeContext runtimeContext) {
Review Comment:
use `runtimeContext.isObjectReuseEnabled()` directly.
##########
hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/utils/RuntimeContextUtils.java:
##########
@@ -55,4 +56,10 @@ public static long getWatermarkInternal(RuntimeContext runtimeContext) {
return Long.parseLong(jobParameters.getOrDefault(AUTO_WATERMARK_INTERVAL.key(),
AUTO_WATERMARK_INTERVAL.defaultValue().toMillis() + ""));
}
+
+ public static boolean isObjectReuseEnabled(RuntimeContext runtimeContext) {
+ Map<String, String> jobParameters = runtimeContext.getGlobalJobParameters();
Review Comment:
use `runtimeContext.isObjectReuseEnabled()` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4257017240
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
}, {
"hash" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827",
"triggerID" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"triggerType" : "PUSH"
}, {
"hash" : "1caf9fa1b4581c353c2a4f0b5a33d419715c679b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13286",
"triggerID" : "1caf9fa1b4581c353c2a4f0b5a33d419715c679b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1caf9fa1b4581c353c2a4f0b5a33d419715c679b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13286)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3849918258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3850099158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #18083:
URL: https://github.com/apache/hudi/pull/18083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4030126535
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/18083?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 45.23%. Comparing base ([`ec04479`](https://app.codecov.io/gh/apache/hudi/commit/ec044790fe71f04710472ad63e7859f1b29e3985?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3cdd7c6`](https://app.codecov.io/gh/apache/hudi/commit/3cdd7c6dfbb34164cd915eb89f746db7a496381a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1028 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (ec04479) and HEAD (3cdd7c6). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (ec04479) | HEAD (3cdd7c6) |
>|------|------|------|
>||1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18083 +/- ##
=============================================
- Coverage 61.43% 45.23% -16.20%
+ Complexity 23082 8419 -14663
=============================================
Files 2108 1186 -922
Lines 127636 60856 -66780
Branches 14534 6492 -8042
=============================================
- Hits 78409 27527 -50882
+ Misses 42873 30342 -12531
+ Partials 6354 2987 -3367
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/18083/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ī | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/18083/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `45.23% <Ćø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 3252 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/18083/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3089986552
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
Review Comment:
Done. Using RuntimeContextUtils.isObjectReuseEnabled(getRuntimeContext()) which adapts across Flink versions. For Flink 1.17-1.20, it delegates to getExecutionConfig().isObjectReuseEnabled(), and for Flink 2.0+/2.1+, it calls runtimeContext.isObjectReuseEnabled() directly.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
Review Comment:
Done. SortKey now copies the normalized key bytes and wraps them as a MemorySegment once in its constructor, so the TreeMap comparator uses pre-computed k1.keySegment/k2.keySegment without any per-comparison allocation.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
+ }
+ }
+
+ @Override
+ public void endInput() {
+ try {
+ // Drain all remaining records and clear buffer
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("EndInput: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during endInput", e);
+ } finally {
Review Comment:
Done. super.endInput() is already outside the try/catch block, consistent with the other append write functions.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
Review Comment:
Done. Added testObjectReuseEnabled() which uses a single reusable GenericRowData instance and verifies all 3 records are distinct after write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2972586832
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
+ }
+ }
+
+ @Override
+ public void endInput() {
+ try {
+ // Drain all remaining records and clear buffer
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("EndInput: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during endInput", e);
+ } finally {
Review Comment:
Let's move `super.endInput();` out of the `finally` clause like other append write functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4099192220
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4099142433
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a4085261407b9451e19565f5f1001f251dc0f7c7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691)
* d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4256414930
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
}, {
"hash" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827",
"triggerID" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"triggerType" : "PUSH"
}, {
"hash" : "1caf9fa1b4581c353c2a4f0b5a33d419715c679b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13286",
"triggerID" : "1caf9fa1b4581c353c2a4f0b5a33d419715c679b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827)
* 1caf9fa1b4581c353c2a4f0b5a33d419715c679b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=13286)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4115597023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722)
* e5921668dfe6aeba6ae085dfd9ae79371e144ae4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4173362965
hey @cshuo: can you re-review the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4029984387
@cshuo Thanks for the thoughtful follow-up. You raise a valid point about the existing `AppendWriteFunctionWithBIMBufferSort` and `AppendWriteFunctionWithDisruptorBufferSort`.
Here's how continuous sorting differs from the existing async approaches:
**Existing approaches (BIM/Disruptor):**
- Batch sort (O(n log n)) at flush time, but move the sort+write to a background thread
- Require double-buffering (2x memory) or a ring buffer
- Add threading complexity (synchronization, error propagation, buffer swaps)
- Sorting still happens as a single O(n log n) burst, just on a different thread
**Continuous sorting (this PR):**
- O(log n) per insert, no batch sort at all
- Single buffer (no double-buffering overhead)
- No threading complexity ā simpler to reason about and debug
- Incremental draining ā when buffer fills, oldest sorted records are written immediately
- Better for latency-sensitive workloads where predictable per-record cost is preferred over throughput
The key trade-off is:
- **BIM/Disruptor** optimize throughput by overlapping sort+write with ingestion (async), at the cost of memory and complexity
- **Continuous sort** optimizes latency predictability by eliminating sort spikes entirely (no batch sort), at the cost of higher per-record overhead (O(log n) vs O(1) insert)
These approaches are complementary ā continuous sorting could potentially be combined with async write in the future. This PR adds it as an opt-in alternative via `write.buffer.sort.continuous.enabled=true`.
I don't have formal benchmark numbers yet. Would it be helpful if I ran a comparison benchmark against the BIM approach to quantify the latency distribution differences?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3089988489
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
Review Comment:
Already final. maxCapacity is `private final long` and drainSize is `private final int`, set once in the constructor.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
Review Comment:
Already done. insertionSequence is reset to 0L alongside sortedRecords.clear() in both snapshotState() and endInput().
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
Review Comment:
The if (cmp != 0) guards are intentional here - this is a 3-level comparator chain (normalized key -> full record -> insertion order). The early returns avoid the more expensive recordComparator.compare() when the cheaper normalizedKeyComputer.compareKey() already distinguishes the records. Keeping the pattern explicit for readability.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctions.java:
##########
Review Comment:
Already handled. The stream pipeline does .map(String::trim).filter(s -> !s.isEmpty()), so "key1, ,key2" splits to ["key1", " ", "key2"], trims to ["key1", "", "key2"], then filters out the empty string to get ["key1", "key2"].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4256405965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
}, {
"hash" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827",
"triggerID" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"triggerType" : "PUSH"
}, {
"hash" : "1caf9fa1b4581c353c2a4f0b5a33d419715c679b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1caf9fa1b4581c353c2a4f0b5a33d419715c679b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827)
* 1caf9fa1b4581c353c2a4f0b5a33d419715c679b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4124958951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
}, {
"hash" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827",
"triggerID" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5921668dfe6aeba6ae085dfd9ae79371e144ae4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796)
* e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4125270441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
}, {
"hash" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827",
"triggerID" : "e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6bc34dd9381dd0e5b3edd1cca06a5895d00fde5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12827)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3857485095
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3849929987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3888308054
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2955402100
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
+ if (sortKeys == null || sortKeys.trim().isEmpty()) {
+ throw new IllegalArgumentException("Sort keys cannot be null or empty for continuous sort");
+ }
+
+ List<String> sortKeyList = Arrays.stream(sortKeys.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+
+ if (sortKeyList.isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Sort keys list is empty after parsing: '%s'", sortKeys));
+ }
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ this.reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the original RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ }
+
+ // Reset for next checkpoint interval
+ sortedRecords.clear();
+ insertionSequence = 0L;
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
+ }
+ }
+
+ @Override
+ public void endInput() {
+ try {
+ // Drain all remaining records
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("EndInput: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ }
+
+ // Clear buffer and reset sequence
+ sortedRecords.clear();
Review Comment:
Done. Moved `sortedRecords.clear()` and `insertionSequence = 0L` inside the `if (!sortedRecords.isEmpty())` branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4084726100
@cshuo All review comments have been addressed:
1. **Sort key resolution**: Now uses `AppendWriteFunctions.resolveSortKeys()` which falls back to record key when `write.buffer.sort.keys` is not set
2. **Object reuse safety**: Added `RowDataSerializer.copy()` when Flink object reuse is enabled to prevent TreeMap ordering corruption
3. **Local variable**: Made `reusableKeyBuffer` a local variable in `open()`
4. **endInput cleanup**: Moved `sortedRecords.clear()` and `insertionSequence = 0L` inside the `if` branch
5. **snapshotState cleanup**: Same as above
6. **Stale class name**: Fixed javadoc reference to `AppendWriteFunctionWithBIMBufferSort`
7. **Config migration**: Removed `WRITE_BUFFER_SORT_CONTINUOUS_ENABLED`, now uses `WRITE_BUFFER_TYPE=CONTINUOUS_SORT` via `BufferType` enum. Updated `AppendWriteFunctions.create()` routing and tests.
Please take another look when you get a chance. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4027933362
hey @prashantwason : can you respond to @cshuo on this asks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2972558964
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
Review Comment:
Let's move `super.snapshotState();` out of `final` clause like other append write functions, since write meta event should not be sent to coordinator when there is ingestion exception.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
Review Comment:
There is compiling error in CI, seems we need to adapt `isObjectReuseEnabled()` for lower flink versions, like flink 1.17, 1.18.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
+ }
+ }
+
+ @Override
+ public void endInput() {
+ try {
+ // Drain all remaining records and clear buffer
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("EndInput: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during endInput", e);
+ } finally {
Review Comment:
Let's move `super.endInput();` out of final clause like other append write functions.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
Review Comment:
The TreeMap comparator wraps keyBytes into new MemorySegment objects for every comparison. Since comparisons happen frequently on insert/drain, this creates avoidable allocation pressure in the hot path, which increases GC preasure.
We can reuse/precompute comparable representations per SortKey, i.e., store wrapped segments in SortKey.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4115957408
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5921668dfe6aeba6ae085dfd9ae79371e144ae4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12796)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4029979841
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
* 3cdd7c6dfbb34164cd915eb89f746db7a496381a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4029988857
Pushed a fix for the CI test failures:
**Fixed 5 validation tests** (`testInvalidDrainSizeNegative`, `testInvalidDrainSizeZero`, `testInvalidBufferSizeZero`, `testInvalidBufferSizeNegative`, `testInvalidSortKeysOnlyWhitespace`, `testInvalidSortKeysOnlyCommas`):
- Moved config validation before `super.open()` which requires a Flink runtime context. Tests that directly instantiate the function without a test harness were getting `IllegalStateException` instead of `IllegalArgumentException`.
**Fixed 2 sort order tests** (`testSortedResult`, `testSortedResultWithContinuousDrain`):
- Added `RecordComparator` for full comparison fallback when normalized keys are equal. Normalized keys are a prefix optimization and may not fully encode all sort fields, so a full `RecordComparator` is needed for correct multi-field sort order.
The Azure CI failure was infrastructure-related (Docker process exit code 1), not a test failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2955402657
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
+ if (sortKeys == null || sortKeys.trim().isEmpty()) {
+ throw new IllegalArgumentException("Sort keys cannot be null or empty for continuous sort");
+ }
+
+ List<String> sortKeyList = Arrays.stream(sortKeys.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+
+ if (sortKeyList.isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Sort keys list is empty after parsing: '%s'", sortKeys));
+ }
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ this.reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the original RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ }
+
+ // Reset for next checkpoint interval
+ sortedRecords.clear();
Review Comment:
Done. Moved `sortedRecords.clear()` and `insertionSequence = 0L` inside the `if (!sortedRecords.isEmpty())` branch.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
Review Comment:
Done. Fixed to reference `AppendWriteFunctionWithBIMBufferSort`.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -724,6 +724,25 @@ public class FlinkOptions extends HoodieConfig {
+ "Data is sorted within the buffer configured by number of records or buffer size. "
+ "The order of entire written file is not guaranteed.");
+ @AdvancedConfig
+ public static final ConfigOption<Boolean> WRITE_BUFFER_SORT_CONTINUOUS_ENABLED = ConfigOptions
+ .key("write.buffer.sort.continuous.enabled")
+ .booleanType()
+ .defaultValue(false) // default use batch sorting
+ .withDescription("Whether to use continuous sorting (TreeMap-based) instead of batch sorting. "
+ + "Continuous sorting provides O(log n) inserts and incremental draining, "
+ + "but has higher per-record overhead. Requires write.buffer.sort.enabled=true.");
Review Comment:
Done. Removed `WRITE_BUFFER_SORT_CONTINUOUS_ENABLED` config entirely. Now uses `WRITE_BUFFER_TYPE=CONTINUOUS_SORT` via the `BufferType` enum. Added `CONTINUOUS_SORT` to `BufferType` and updated `AppendWriteFunctions.create()` to route accordingly. Updated tests as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4099171204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a4085261407b9451e19565f5f1001f251dc0f7c7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691)
* d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2972586832
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
+ }
+ }
+
+ @Override
+ public void endInput() {
+ try {
+ // Drain all remaining records and clear buffer
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("EndInput: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during endInput", e);
+ } finally {
Review Comment:
Let's move `super.endInput();` out of the finally clause like other append write functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2973259107
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
Review Comment:
Besides, can you also add a test with object reuse enabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3857509560
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3095504311
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/append/ITTestAppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.flink.table.types.logical.IntType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.TimestampType;
+import org.apache.flink.table.types.logical.VarCharType;
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.sink.buffer.BufferType;
+import org.apache.hudi.sink.utils.TestWriteBase;
+import org.apache.hudi.utils.TestConfigurations;
+import org.apache.hudi.utils.TestData;
+
+import org.apache.avro.generic.GenericRecord;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.data.TimestampData;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.io.File;
+import java.sql.Timestamp;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Test cases for {@link AppendWriteFunctionWithContinuousSort}.
+ */
+public class ITTestAppendWriteFunctionWithContinuousSort extends TestWriteBase {
+ private Configuration conf;
+ private RowType rowType;
+
+ @TempDir
+ protected File tempFile;
Review Comment:
_ā ļø Potential issue_ | _š” Minor_
**Shadowed `@TempDir tempFile` field.**
This class declares `@TempDir protected File tempFile` (line 61-62), but `TestWriteBase` already declares `@TempDir protected File tempFile` (inherited). This shadowing may cause confusion and could lead to unexpected behavior if both are initialized differently by JUnit.
Consider removing the local declaration since the inherited field should suffice.
<details>
<summary>š§ Proposed fix</summary>
```diff
public class ITTestAppendWriteFunctionWithContinuousSort extends TestWriteBase {
private Configuration conf;
private RowType rowType;
-
- `@TempDir`
- protected File tempFile;
```
</details>
<details>
<summary>š¤ Prompt for AI Agents</summary>
```
Verify each finding against the current code and only fix it if needed.
In
`@hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/sink/append/ITTestAppendWriteFunctionWithContinuousSort.java`
around lines 61 - 62, The class ITTestAppendWriteFunctionWithContinuousSort
currently redeclares the JUnit TempDir field as "protected File tempFile",
shadowing the same "@TempDir protected File tempFile" declared in the superclass
TestWriteBase; remove the local declaration of tempFile from
ITTestAppendWriteFunctionWithContinuousSort so the test uses the inherited
TestWriteBase.tempFile, and update any local references in that class to rely on
the inherited field name (no other changes required).
```
</details>
<!-- fingerprinting:phantom:medusa:ocelot:ceaaf6c9-cefb-48fa-8f1c-105371993f70 -->
<!-- This is an auto-generated comment by CodeRabbit -->
ā *CodeRabbit* ([original](https://github.com/yihua/hudi/pull/46#discussion_r3095501107)) (source:comment#3095501107)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3036444773
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ int cmp = normalizedKeyComputer.compareKey(k1.keySegment, 0, k2.keySegment, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = RuntimeContextUtils.isObjectReuseEnabled(getRuntimeContext());
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
Review Comment:
š¤ Unlike `BIMBufferSort`, this sorting mode does not bound its memory footprint using `write.task.max.size`.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ int cmp = normalizedKeyComputer.compareKey(k1.keySegment, 0, k2.keySegment, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = RuntimeContextUtils.isObjectReuseEnabled(getRuntimeContext());
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
Review Comment:
š¤ Allocating an Iterator for every drained record can create unnecessary garbage when drainSize is small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3888315765
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3888306409
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4030008273
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
* 3cdd7c6dfbb34164cd915eb89f746db7a496381a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r2911817445
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
+ if (sortKeys == null || sortKeys.trim().isEmpty()) {
+ throw new IllegalArgumentException("Sort keys cannot be null or empty for continuous sort");
+ }
+
+ List<String> sortKeyList = Arrays.stream(sortKeys.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+
+ if (sortKeyList.isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Sort keys list is empty after parsing: '%s'", sortKeys));
+ }
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ this.reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
Review Comment:
The map stores raw RowData references and later compares them. With Flink object reuse enabled, those retained rows can be mutated after insertion, which breaks
TreeMap ordering guarantees and risks incorrect output.
Use `getRuntimeContext().isObjectReuseEnabled();` to check whether object reuse is enabled, and copy rowData here if necessary.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
Review Comment:
Class name `AppendWriteFunctionWithBufferSort` is stale and has been renamed.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
+ if (sortKeys == null || sortKeys.trim().isEmpty()) {
+ throw new IllegalArgumentException("Sort keys cannot be null or empty for continuous sort");
+ }
+
+ List<String> sortKeyList = Arrays.stream(sortKeys.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+
+ if (sortKeyList.isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Sort keys list is empty after parsing: '%s'", sortKeys));
+ }
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ this.reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the original RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ }
+
+ // Reset for next checkpoint interval
+ sortedRecords.clear();
Review Comment:
Put this line inside the upper if branch?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
Review Comment:
can be local variable.
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
Review Comment:
The existing append-sort code resolves missing write.buffer.sort.keys by falling back to the record key via AppendWriteFunctions.resolveSortKeys, can the function also follow the behavior?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -724,6 +724,25 @@ public class FlinkOptions extends HoodieConfig {
+ "Data is sorted within the buffer configured by number of records or buffer size. "
+ "The order of entire written file is not guaranteed.");
+ @AdvancedConfig
+ public static final ConfigOption<Boolean> WRITE_BUFFER_SORT_CONTINUOUS_ENABLED = ConfigOptions
+ .key("write.buffer.sort.continuous.enabled")
+ .booleanType()
+ .defaultValue(false) // default use batch sorting
+ .withDescription("Whether to use continuous sorting (TreeMap-based) instead of batch sorting. "
+ + "Continuous sorting provides O(log n) inserts and incremental draining, "
+ + "but has higher per-record overhead. Requires write.buffer.sort.enabled=true.");
Review Comment:
`write.buffer.sort.enabled` is deprecated, and append writing uses `WRITE_BUFFER_TYPE` to select different write functions, can we also use `WRITE_BUFFER_TYPE` here?
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient byte[] reusableKeyBuffer;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Parse and validate sort keys
+ String sortKeys = config.get(FlinkOptions.WRITE_BUFFER_SORT_KEYS);
+ if (sortKeys == null || sortKeys.trim().isEmpty()) {
+ throw new IllegalArgumentException("Sort keys cannot be null or empty for continuous sort");
+ }
+
+ List<String> sortKeyList = Arrays.stream(sortKeys.split(","))
+ .map(String::trim)
+ .filter(s -> !s.isEmpty())
+ .collect(Collectors.toList());
+
+ if (sortKeyList.isEmpty()) {
+ throw new IllegalArgumentException(
+ String.format("Sort keys list is empty after parsing: '%s'", sortKeys));
+ }
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ this.reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the original RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ }
+
+ // Reset for next checkpoint interval
+ sortedRecords.clear();
+ insertionSequence = 0L;
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
+ }
+ }
+
+ @Override
+ public void endInput() {
+ try {
+ // Drain all remaining records
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("EndInput: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ }
+
+ // Clear buffer and reset sequence
+ sortedRecords.clear();
Review Comment:
Put this line inside the upper if branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-3949614388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 8226525e5f4f9bc75d61a4d24ab249f59db9b43a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3089989034
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ int cmp = normalizedKeyComputer.compareKey(k1.keySegment, 0, k2.keySegment, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = RuntimeContextUtils.isObjectReuseEnabled(getRuntimeContext());
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
Review Comment:
Fixed. Added TotalSizeTracer to bound the buffer memory footprint using write.task.max.size. Now processElement() triggers draining when either the record count limit (write.buffer.size) or memory size limit is exceeded. Record size is estimated once via ObjectSizeCalculator and tracked via TotalSizeTracer.trace()/countDown()/reset().
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+import org.apache.hudi.utils.RuntimeContextUtils;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ int cmp = normalizedKeyComputer.compareKey(k1.keySegment, 0, k2.keySegment, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = RuntimeContextUtils.isObjectReuseEnabled(getRuntimeContext());
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
Review Comment:
Fixed. Replaced the Iterator-based loop with TreeMap.pollFirstEntry() which removes and returns the first entry directly, avoiding Iterator object allocation entirely. This matters especially when drainSize=1 (the default), where each incoming record at capacity would have created a new Iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3089985336
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,330 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ MemorySegment seg1 = MemorySegmentFactory.wrap(k1.keyBytes);
+ MemorySegment seg2 = MemorySegmentFactory.wrap(k2.keyBytes);
+
+ int cmp = normalizedKeyComputer.compareKey(seg1, 0, seg2, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().isObjectReuseEnabled();
+ if (this.objectReuseEnabled) {
+ this.rowDataSerializer = new RowDataSerializer(rowType);
+ }
+
+ // Initialize metrics
+ this.totalDrainOperations = 0;
+ this.totalDrainedRecords = 0;
+ this.totalInserted = 0;
+
+ LOG.info("AppendWriteFunctionWithContinuousSort initialized successfully");
+ }
+
+ @Override
+ public void processElement(T value, Context ctx, Collector<RowData> out) throws Exception {
+ RowData data = (RowData) value;
+
+ // Check if buffer has reached max capacity
+ if (sortedRecords.size() >= maxCapacity) {
+ drainRecords(drainSize);
+
+ // Verify there's space after draining
+ if (sortedRecords.size() >= maxCapacity) {
+ throw new HoodieException(
+ String.format("Buffer cannot accept record after draining. "
+ + "Buffer size: %d, maxCapacity: %d, drainSize: %d",
+ sortedRecords.size(), maxCapacity, drainSize));
+ }
+ }
+
+ // Copy RowData when object reuse is enabled to prevent mutation after insertion
+ if (objectReuseEnabled) {
+ data = rowDataSerializer.copy(data);
+ }
+
+ // Write to buffer (maintains sorted order)
+ // Compute normalized key into reusable segment
+ normalizedKeyComputer.putKey(data, reusableKeySegment, 0);
+
+ // Create sort key (copies the normalized key from reusable segment)
+ SortKey key = new SortKey(reusableKeySegment, normalizedKeySize, data, insertionSequence++);
+
+ // Store the RowData
+ sortedRecords.put(key, data);
+
+ totalInserted++;
+ }
+
+ /**
+ * Drain oldest records from buffer and write to storage.
+ */
+ private void drainRecords(int count) throws IOException {
+ if (sortedRecords.isEmpty()) {
+ return;
+ }
+
+ // Initialize writer if needed
+ if (this.writerHelper == null) {
+ initWriterHelper();
+ }
+
+ // Drain records from TreeMap
+ int actualCount = Math.min(count, sortedRecords.size());
+ int drained = 0;
+
+ Iterator<Map.Entry<SortKey, RowData>> iterator = sortedRecords.entrySet().iterator();
+ while (iterator.hasNext() && drained < actualCount) {
+ Map.Entry<SortKey, RowData> entry = iterator.next();
+ RowData record = entry.getValue();
+
+ // Write record
+ writerHelper.write(record);
+
+ // Remove from TreeMap - memory immediately reclaimed
+ iterator.remove();
+ drained++;
+ }
+
+ totalDrainOperations++;
+ totalDrainedRecords += drained;
+ }
+
+ @Override
+ public void snapshotState() {
+ try {
+ // Drain all remaining records and reset for next checkpoint interval
+ if (!sortedRecords.isEmpty()) {
+ LOG.info("Snapshot: draining {} remaining records", sortedRecords.size());
+ drainRecords(sortedRecords.size());
+ sortedRecords.clear();
+ insertionSequence = 0L;
+ }
+
+ LOG.info("Snapshot complete: total drained={}, operations={}",
+ totalDrainedRecords, totalDrainOperations);
+
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to drain buffer during snapshot", e);
+ } finally {
+ super.snapshotState();
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #18083:
URL: https://github.com/apache/hudi/pull/18083#discussion_r3089987119
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/append/AppendWriteFunctionWithContinuousSort.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.sink.append;
+
+import org.apache.hudi.configuration.FlinkOptions;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.sink.StreamWriteOperatorCoordinator;
+import org.apache.hudi.sink.bulk.sort.SortOperatorGen;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.planner.codegen.sort.SortCodeGenerator;
+import org.apache.flink.table.runtime.typeutils.RowDataSerializer;
+import org.apache.flink.table.runtime.generated.GeneratedNormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.GeneratedRecordComparator;
+import org.apache.flink.table.runtime.generated.NormalizedKeyComputer;
+import org.apache.flink.table.runtime.generated.RecordComparator;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.util.Collector;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+/**
+ * Sink function to write data with continuous sorting for improved compression.
+ *
+ * <p>Unlike {@link AppendWriteFunctionWithBIMBufferSort} which uses batch sorting,
+ * this function maintains sorted order continuously using a TreeMap, providing:
+ * <ul>
+ * <li>Non-blocking inserts (O(log n) vs O(1) + periodic O(n log n))</li>
+ * <li>Incremental draining without re-sorting</li>
+ * <li>Predictable latency (no sort spikes)</li>
+ * </ul>
+ *
+ * <p>Strategy:
+ * <ol>
+ * <li>Records are inserted in sorted order (TreeMap)</li>
+ * <li>When buffer reaches max capacity, oldest record(s) are drained synchronously</li>
+ * <li>Drain size is configurable to balance latency vs. throughput vs compression ratio</li>
+ * </ol>
+ *
+ * @param <T> Type of the input record
+ * @see StreamWriteOperatorCoordinator
+ */
+public class AppendWriteFunctionWithContinuousSort<T> extends AppendWriteFunction<T> {
+
+ private static final Logger LOG = LoggerFactory.getLogger(AppendWriteFunctionWithContinuousSort.class);
+
+ private final long maxCapacity;
+ private final int drainSize;
+
+ private transient TreeMap<SortKey, RowData> sortedRecords;
+ private transient long insertionSequence;
+
+ // Sort key computation
+ private transient NormalizedKeyComputer normalizedKeyComputer;
+ private transient RecordComparator recordComparator;
+ private transient MemorySegment reusableKeySegment;
+ private transient int normalizedKeySize;
+ private transient boolean objectReuseEnabled;
+ private transient RowDataSerializer rowDataSerializer;
+
+ // Metrics
+ private transient long totalDrainOperations;
+ private transient long totalDrainedRecords;
+ private transient long totalInserted;
+
+ public AppendWriteFunctionWithContinuousSort(Configuration config, RowType rowType) {
+ super(config, rowType);
+
+ // Configuration
+ this.maxCapacity = config.get(FlinkOptions.WRITE_BUFFER_SIZE);
+ this.drainSize = config.get(FlinkOptions.WRITE_BUFFER_SORT_CONTINUOUS_DRAIN_SIZE);
+
+ LOG.info("AppendWriteFunctionWithContinuousSort created: maxCapacity={}, drainSize={}",
+ maxCapacity, drainSize);
+ }
+
+ @Override
+ public void open(Configuration parameters) throws Exception {
+ // Validate configuration before calling super.open() which requires Flink runtime context
+ if (maxCapacity <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Buffer capacity must be positive, got: %d", maxCapacity));
+ }
+
+ if (drainSize <= 0) {
+ throw new IllegalArgumentException(
+ String.format("Drain size must be positive, got: %d", drainSize));
+ }
+
+ // Resolve sort keys, falling back to record key if not specified
+ List<String> sortKeyList = AppendWriteFunctions.resolveSortKeys(config);
+
+ super.open(parameters);
+
+ LOG.info("Initializing continuous sort with keys: {}", sortKeyList);
+
+ // Create sort code generator for normalized key computation and record comparison
+ SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, sortKeyList.toArray(new String[0]));
+ SortCodeGenerator codeGenerator = sortOperatorGen.createSortCodeGenerator();
+ GeneratedNormalizedKeyComputer generatedKeyComputer = codeGenerator.generateNormalizedKeyComputer("ContinuousSortKeyComputer");
+ GeneratedRecordComparator generatedComparator = codeGenerator.generateRecordComparator("ContinuousSortComparator");
+
+ // Instantiate code-generated components
+ ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
+ this.normalizedKeyComputer = generatedKeyComputer.newInstance(classLoader);
+ this.recordComparator = generatedComparator.newInstance(classLoader);
+ this.normalizedKeySize = normalizedKeyComputer.getNumKeyBytes();
+
+ // Initialize TreeMap with comparator that uses normalized keys for fast comparison
+ // and falls back to RecordComparator for full comparison when normalized keys are equal
+ this.sortedRecords = new TreeMap<>((k1, k2) -> {
+ int cmp = normalizedKeyComputer.compareKey(k1.keySegment, 0, k2.keySegment, 0);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Normalized keys are equal - use full record comparison for correct ordering
+ cmp = recordComparator.compare(k1.record, k2.record);
+ if (cmp != 0) {
+ return cmp;
+ }
+ // Records are equal by sort keys - use insertion order for stability
+ return Long.compare(k1.insertionOrder, k2.insertionOrder);
+ });
+ this.insertionSequence = 0L;
+
+ // Allocate reusable on-heap buffer for computing keys
+ byte[] reusableKeyBuffer = new byte[normalizedKeySize];
+ this.reusableKeySegment = MemorySegmentFactory.wrap(reusableKeyBuffer);
+
+ // Detect object reuse mode and create serializer for copying if needed
+ this.objectReuseEnabled = getRuntimeContext().getExecutionConfig().isObjectReuseEnabled();
Review Comment:
Done. Using RuntimeContextUtils.isObjectReuseEnabled() which delegates to getExecutionConfig().isObjectReuseEnabled() for Flink 1.17-1.20 and runtimeContext.isObjectReuseEnabled() directly for Flink 2.0+/2.1+.
##########
hudi-flink-datasource/hudi-flink2.0.x/src/main/java/org/apache/hudi/utils/RuntimeContextUtils.java:
##########
@@ -55,4 +56,10 @@ public static long getWatermarkInternal(RuntimeContext runtimeContext) {
return Long.parseLong(jobParameters.getOrDefault(AUTO_WATERMARK_INTERVAL.key(),
AUTO_WATERMARK_INTERVAL.defaultValue().toMillis() + ""));
}
+
+ public static boolean isObjectReuseEnabled(RuntimeContext runtimeContext) {
Review Comment:
Done. Using runtimeContext.isObjectReuseEnabled() directly for Flink 2.0.x.
##########
hudi-flink-datasource/hudi-flink2.1.x/src/main/java/org/apache/hudi/utils/RuntimeContextUtils.java:
##########
@@ -55,4 +56,10 @@ public static long getWatermarkInternal(RuntimeContext runtimeContext) {
return Long.parseLong(jobParameters.getOrDefault(AUTO_WATERMARK_INTERVAL.key(),
AUTO_WATERMARK_INTERVAL.defaultValue().toMillis() + ""));
}
+
+ public static boolean isObjectReuseEnabled(RuntimeContext runtimeContext) {
+ Map<String, String> jobParameters = runtimeContext.getGlobalJobParameters();
Review Comment:
Done. Using runtimeContext.isObjectReuseEnabled() directly for Flink 2.1.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4265019674
+1
cc @nsivabalan @danny0405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(flink): Implement continuous sorting feature for append write [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #18083:
URL: https://github.com/apache/hudi/pull/18083#issuecomment-4115586292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"triggerType" : "PUSH"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3857485095",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3888306409",
"triggerType" : "MANUAL"
}, {
"hash" : "8226525e5f4f9bc75d61a4d24ab249f59db9b43a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=11664",
"triggerID" : "3949598053",
"triggerType" : "MANUAL"
}, {
"hash" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12529",
"triggerID" : "3cdd7c6dfbb34164cd915eb89f746db7a496381a",
"triggerType" : "PUSH"
}, {
"hash" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12691",
"triggerID" : "a4085261407b9451e19565f5f1001f251dc0f7c7",
"triggerType" : "PUSH"
}, {
"hash" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722",
"triggerID" : "d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db",
"triggerType" : "PUSH"
}, {
"hash" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e5921668dfe6aeba6ae085dfd9ae79371e144ae4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d1d2d4f8325d936f82a4f88bc3ce5b9fb11ce1db Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=12722)
* e5921668dfe6aeba6ae085dfd9ae79371e144ae4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org