You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "dybyte (via GitHub)" <gi...@apache.org> on 2026/01/26 12:46:36 UTC
[PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
dybyte opened a new pull request, #10399:
URL: https://github.com/apache/seatunnel/pull/10399
Fixes: https://github.com/apache/seatunnel/issues/10329
### Purpose of this pull request
This PR introduces LSM-style write and compaction logic for IMap external storage, ensuring
sorted immutable files, automatic recovery from temporary files, and size-based compaction.
<img width="623" height="600" alt="screenshot" src="https://github.com/user-attachments/assets/58c30bff-0b67-4b96-835a-d7bb8bdc479b" />
### Write & Compaction Workflow
1. During `write`, data is first appended to a temporary file in an unsorted format.
2. When the size of the temporary file exceeds `blockSize`, a `sortFlush` is triggered:
- Records in the temporary file are sorted by key.
- The sorted data is written into a new immutable data file.
- The temporary file is then deleted.
3. The system tracks `totalBytes` of persisted files. When `totalBytes` exceeds `compactionThreshold`, compaction is triggered.
4. During compaction, smaller files are merged first to reduce file fragmentation, and only the latest record per key is retained.
5. On `initialize`, if any temporary files are detected (for example, due to an unexpected shutdown), `sortFlush` is executed to ensure:
- All remaining temporary data is flushed into sorted files.
- No temporary files are left behind.
As a result, only sorted immutable files remain before normal operation resumes.
```
Write
↓
Temp File
↓ (blockSize)
SortFlush
↓
Data File
↓ (totalBytes > threshold)
Compaction
```
### Does this PR introduce _any_ user-facing change?
Yes. Please refer to docs.
### How was this patch tested?
- `LSMWriterTest`(parameterized for both Cloud and HDFS LSMWriters)
- `IMapFileStorageTest`
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-4556759277
Hi @davidzollo . PTAL when you have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2740407029
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/disruptor/WALCompactionDisruptor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.seatunnel.engine.imap.storage.file.disruptor;
+
+import org.apache.seatunnel.engine.imap.storage.api.exception.IMapStorageException;
+import org.apache.seatunnel.engine.imap.storage.file.bean.IMapFileData;
+import org.apache.seatunnel.engine.imap.storage.file.common.WALLSMWriter;
+import org.apache.seatunnel.engine.imap.storage.file.common.WALWriter;
+import org.apache.seatunnel.engine.imap.storage.file.config.FileConfiguration;
+import org.apache.seatunnel.engine.serializer.api.Serializer;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.EventTranslatorOneArg;
+import com.lmax.disruptor.TimeoutException;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import com.lmax.disruptor.util.DaemonThreadFactory;
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+
+@Slf4j
+public class WALCompactionDisruptor extends AbstractWALDisruptor {
+ private static final EventTranslatorOneArg<FileWALEvent, WALEventType> COMPACTION_TRANSLATOR =
+ (event, sequence, status) -> {
+ event.setData(null);
+ event.setType(status);
+ event.setRequestId(0L);
+ };
+
+ private Disruptor<FileWALEvent> compactionDisruptor;
+
+ public WALCompactionDisruptor(
+ FileSystem fs,
+ FileConfiguration fileConfiguration,
+ String parentPath,
+ Serializer serializer,
+ Map<String, Object> config) {
+ ThreadFactory threadFactory = DaemonThreadFactory.INSTANCE;
+ this.disruptor =
+ new Disruptor<>(
+ FileWALEvent.FACTORY,
+ DEFAULT_RING_BUFFER_SIZE,
+ threadFactory,
+ ProducerType.SINGLE,
+ new BlockingWaitStrategy());
+
+ WALWriter writer;
+ try {
+ writer =
+ new WALLSMWriter(
+ fs, fileConfiguration, new Path(parentPath), serializer, config);
+ } catch (IOException e) {
+ throw new IMapStorageException(
+ e, "create new current writer failed, parent path is %s", parentPath);
+ }
+
+ disruptor.handleEventsWithWorkerPool(new WALWorkHandler(writer));
+
+ disruptor.start();
+
+ this.compactionDisruptor =
+ new Disruptor<>(
+ FileWALEvent.FACTORY,
+ DEFAULT_RING_BUFFER_SIZE,
+ threadFactory,
+ ProducerType.SINGLE,
+ new BlockingWaitStrategy());
+
+ compactionDisruptor.handleEventsWithWorkerPool(
+ new WALCompactionWorkHandler((WALLSMWriter) writer));
+
+ compactionDisruptor.start();
+ }
+
+ @Override
+ public boolean tryPublish(IMapFileData message, WALEventType status, long requestId) {
+ if (isClosed()) {
+ return false;
+ }
+ disruptor.getRingBuffer().publishEvent(TRANSLATOR, message, status, requestId);
+ compactionDisruptor.getRingBuffer().publishEvent(COMPACTION_TRANSLATOR, status);
+ return true;
Review Comment:
I have created the compaction scheduler in `AbstractLSMWriter` to decouple it from the write path, as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-4196248958
I re-checked the latest delta locally.
The specific blocker I raised earlier still remains fixed:
- `IMapFileStorage.initialize()` now parses `compaction.imap` when the config string is present (`IMapFileStorage.java:169-182`), so the compaction-enabled path is actually reachable
- the latest test update is consistent with that contract by passing `COMPACTION_ENABLED = "true"` (`IMapFileStorageTest.java:61-70`)
I do not see a new blocker in the latest change I re-checked from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-3802910088
<!-- code-pr-reviewer -->
<!-- cpr:pr_reply_v2_parts {"group": "apache/seatunnel#10399", "part": 1, "total": 1} -->
### Issue 1: WALCompactionDisruptor publishes complete events repeatedly
**Location**: `WALCompactionDisruptor.java:98-99`
**Severity**: MAJOR
**Category**: Performance/Design
**Issue**: Publishing complete events to two RingBuffers on every write increases memory overhead; compaction threads do not need complete data
**Impact**: Increased memory usage in high-concurrency scenarios, potentially affecting performance
### Issue 2: CloudLSMWriter frequently creates file streams
**Location**: `CloudLSMWriter.java:77`
**Severity**: MAJOR
**Category**: Performance
**Issue**: Creating a new FSDataOutputStream on every write() call, which is unfriendly to cloud storage
**Impact**: Decreased write performance in cloud storage scenarios, increased number of API calls
### Issue 3: Concurrent access to writeBatch and bf is unprotected
**Location**: `AbstractLSMWriter.java:68-69, 128-151`
**Severity**: CRITICAL
**Category**: Concurrency Safety
**Issue**: `writeBatch` and `bf` are accessed concurrently by multiple threads without complete synchronization
**Impact**: May lead to `ConcurrentModificationException` or data corruption
### Issue 4: compaction() method lacks synchronization protection
**Location**: `AbstractLSMWriter.java:164-200`
**Severity**: MAJOR
**Category**: Concurrency Safety
**Issue**: Race condition exists between `compaction()` and `sortFlush`
**Impact**: May lead to inconsistent file state
### Issue 5: Delete markers not cleaned up during compaction
**Location**: `AbstractLSMWriter.java:232-235`
**Severity**: MAJOR
**Category**: Data Consistency
**Issue**: Delete markers (tombstone records) remain after compaction, not thoroughly cleaned up
**Impact**: Accumulation of tombstone records after long-term operation, wasting storage and affecting query performance
### Issue 6: Resource cleanup in exception handling is incomplete
**Location**: `CloudLSMWriter.java:95-105`
**Severity**: MINOR
**Category**: Error Handling
**Issue**: `bf` not cleaned up when exceptions occur
**Impact**: May lead to data duplication
### Issue 7: Insufficient concurrency test coverage
**Location**: `LSMWriterTest.java:327-396`
**Severity**: MINOR
**Category**: Test Coverage
**Issue**: Insufficient test intensity for concurrency scenarios
**Impact**: Concurrency bugs under edge conditions may go undetected in testing
### Issue 8: CompactionFile comparison logic is incomplete
**Location**: `CompactionFile.java:40-54`
**Severity**: MINOR
**Category**: Code Quality
**Issue**: `compareTo`/`equals`/`hashCode` are inconsistent
**Impact**: May lead to abnormal `PriorityBlockingQueue` behavior
### Issue 9: Configuration error handling is not user-friendly
**Location**: `IMapFileStorage.java:174-186`
**Severity**: MINOR
**Category**: User Experience
**Issue**: JSON parsing error messages are not clear enough
**Impact**: Difficult to troubleshoot when configuration errors occur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@seatunnel.apache.org>.
github-actions[bot] commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-6933891814
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2740411494
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/bean/IMapFileData.java:
##########
@@ -27,6 +27,8 @@
import java.io.Serializable;
+import static org.apache.hadoop.hdfs.DFSUtilClient.compareBytes;
Review Comment:
Since there was no appropriate JDK method, I added a replacement in `WALDataUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-4191620337
@dybyte I re-checked the latest HEAD, and the specific blocker from my last review is fixed now.
`IMapFileStorage.initialize()` now parses `compaction.imap` when the config string is present (`IMapFileStorage.java:169-173`), so `compaction.imap = "true"` can actually drive the `WALDisruptor(..., configuration)` path (`IMapFileStorage.java:178-182`). The earlier flag inversion is no longer there.
So that particular issue is resolved from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2736514178
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/disruptor/WALCompactionDisruptor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.seatunnel.engine.imap.storage.file.disruptor;
+
+import org.apache.seatunnel.engine.imap.storage.api.exception.IMapStorageException;
+import org.apache.seatunnel.engine.imap.storage.file.bean.IMapFileData;
+import org.apache.seatunnel.engine.imap.storage.file.common.WALLSMWriter;
+import org.apache.seatunnel.engine.imap.storage.file.common.WALWriter;
+import org.apache.seatunnel.engine.imap.storage.file.config.FileConfiguration;
+import org.apache.seatunnel.engine.serializer.api.Serializer;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import com.lmax.disruptor.BlockingWaitStrategy;
+import com.lmax.disruptor.EventTranslatorOneArg;
+import com.lmax.disruptor.TimeoutException;
+import com.lmax.disruptor.dsl.Disruptor;
+import com.lmax.disruptor.dsl.ProducerType;
+import com.lmax.disruptor.util.DaemonThreadFactory;
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+
+@Slf4j
+public class WALCompactionDisruptor extends AbstractWALDisruptor {
+ private static final EventTranslatorOneArg<FileWALEvent, WALEventType> COMPACTION_TRANSLATOR =
+ (event, sequence, status) -> {
+ event.setData(null);
+ event.setType(status);
+ event.setRequestId(0L);
+ };
+
+ private Disruptor<FileWALEvent> compactionDisruptor;
+
+ public WALCompactionDisruptor(
+ FileSystem fs,
+ FileConfiguration fileConfiguration,
+ String parentPath,
+ Serializer serializer,
+ Map<String, Object> config) {
+ ThreadFactory threadFactory = DaemonThreadFactory.INSTANCE;
+ this.disruptor =
+ new Disruptor<>(
+ FileWALEvent.FACTORY,
+ DEFAULT_RING_BUFFER_SIZE,
+ threadFactory,
+ ProducerType.SINGLE,
+ new BlockingWaitStrategy());
+
+ WALWriter writer;
+ try {
+ writer =
+ new WALLSMWriter(
+ fs, fileConfiguration, new Path(parentPath), serializer, config);
+ } catch (IOException e) {
+ throw new IMapStorageException(
+ e, "create new current writer failed, parent path is %s", parentPath);
+ }
+
+ disruptor.handleEventsWithWorkerPool(new WALWorkHandler(writer));
+
+ disruptor.start();
+
+ this.compactionDisruptor =
+ new Disruptor<>(
+ FileWALEvent.FACTORY,
+ DEFAULT_RING_BUFFER_SIZE,
+ threadFactory,
+ ProducerType.SINGLE,
+ new BlockingWaitStrategy());
+
+ compactionDisruptor.handleEventsWithWorkerPool(
+ new WALCompactionWorkHandler((WALLSMWriter) writer));
+
+ compactionDisruptor.start();
+ }
+
+ @Override
+ public boolean tryPublish(IMapFileData message, WALEventType status, long requestId) {
+ if (isClosed()) {
+ return false;
+ }
+ disruptor.getRingBuffer().publishEvent(TRANSLATOR, message, status, requestId);
+ compactionDisruptor.getRingBuffer().publishEvent(COMPACTION_TRANSLATOR, status);
+ return true;
Review Comment:
The current implementation couples the compaction trigger directly with the write path in a 1:1 ratio. In `WALCompactionDisruptor.tryPublish()`, **every single write request** publishes a corresponding event to the `compactionDisruptor`. This causes the `WALCompactionWorkHandler` to invoke the `writer.compaction()` method for every single record written to the WAL.
Although the `compaction()` method performs a check (`totalBytes < threshold`) to return early, this design is fundamentally flawed for high-throughput scenarios:
* **Event Flooding**: A bulk write of 100,000 records will trigger 100,000 unnecessary calls to the compaction logic.
* **Resource Waste**: The compaction thread is forced into a "busy-check" loop, consuming CPU cycles to check thresholds repeatedly instead of sleeping until needed.
* **Coupling**: Compaction is a heavy io maintenance task and should not be driven by the frequency of incoming data ingestion.
**Recommendation**:
Since compaction is a heavy I/O operation, it must be decoupled from the high-frequency write event stream. I think you can implement **multiple trigger strategies** to make it robust:
1. **Scheduled Trigger (Time-based)**: Use a `ScheduledExecutorService` to trigger the compaction check periodically (e.g., every 60 seconds). This is the most predictable method.
2. **Active Threshold Check**: Trigger only when the `totalBytes` counter crosses a specific watermark, rather than checking on every increment.
Remove the `compactionDisruptor` event publishing from `tryPublish`. Initialize a background scheduler in `IMapFileStorage` or `AbstractLSMWriter` that invokes `compaction()` at a configurable interval (`compactionInterval`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2740410741
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/common/WALLSMWriter.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.seatunnel.engine.imap.storage.file.common;
+
+import org.apache.seatunnel.engine.imap.storage.file.config.FileConfiguration;
+import org.apache.seatunnel.engine.imap.storage.file.wal.DiscoveryWalFileFactory;
+import org.apache.seatunnel.engine.serializer.api.Serializer;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Map;
+
+public class WALLSMWriter extends WALWriter {
+ public WALLSMWriter(
+ FileSystem fs,
+ FileConfiguration fileConfiguration,
+ Path parentPath,
+ Serializer serializer,
+ Map<String, Object> config)
+ throws IOException {
+ super(fs, fileConfiguration, parentPath, serializer);
+ this.writer = DiscoveryWalFileFactory.getLSMWriter(fileConfiguration.getName(), config);
+ this.writer.setBlockSize(fileConfiguration.getConfiguration().getBlockSize());
+ this.writer.initialize(fs, parentPath, serializer);
Review Comment:
You are right, I hadn't considered the potential resource leak from creating the writer in the superclass. I have now fixed this issue to ensure proper cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2736748286
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/common/WALLSMWriter.java:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.seatunnel.engine.imap.storage.file.common;
+
+import org.apache.seatunnel.engine.imap.storage.file.config.FileConfiguration;
+import org.apache.seatunnel.engine.imap.storage.file.wal.DiscoveryWalFileFactory;
+import org.apache.seatunnel.engine.serializer.api.Serializer;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Map;
+
+public class WALLSMWriter extends WALWriter {
+ public WALLSMWriter(
+ FileSystem fs,
+ FileConfiguration fileConfiguration,
+ Path parentPath,
+ Serializer serializer,
+ Map<String, Object> config)
+ throws IOException {
+ super(fs, fileConfiguration, parentPath, serializer);
+ this.writer = DiscoveryWalFileFactory.getLSMWriter(fileConfiguration.getName(), config);
+ this.writer.setBlockSize(fileConfiguration.getConfiguration().getBlockSize());
+ this.writer.initialize(fs, parentPath, serializer);
Review Comment:
Why is super called and then re-created? The writer created in super may not be closed, leading to resource leakage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2736804603
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/wal/writer/lsm/AbstractLSMWriter.java:
##########
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.seatunnel.engine.imap.storage.file.wal.writer.lsm;
+
+import org.apache.seatunnel.engine.imap.storage.api.exception.IMapStorageException;
+import org.apache.seatunnel.engine.imap.storage.file.bean.IMapFileData;
+import org.apache.seatunnel.engine.imap.storage.file.common.FileConstants;
+import org.apache.seatunnel.engine.imap.storage.file.common.WALDataUtils;
+import org.apache.seatunnel.engine.imap.storage.file.wal.IMapFileIterator;
+import org.apache.seatunnel.engine.imap.storage.file.wal.WALFileIterator;
+import org.apache.seatunnel.engine.imap.storage.file.wal.reader.DefaultReader;
+import org.apache.seatunnel.engine.imap.storage.file.wal.writer.CompactionFile;
+import org.apache.seatunnel.engine.imap.storage.file.wal.writer.IFileWriter;
+import org.apache.seatunnel.engine.serializer.api.Serializer;
+
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import lombok.extern.slf4j.Slf4j;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.atomic.AtomicLong;
+
+@Slf4j
+public abstract class AbstractLSMWriter implements IFileWriter<IMapFileData> {
+
+ protected final AtomicLong index = new AtomicLong(0);
+ protected final AtomicLong tmpIndex = new AtomicLong(0);
+ protected long blockSize = 1L * 1024 * 1024;
+ protected long compactionThreshold = 512L * 1024 * 1024;
+ protected long maxSingleFileSize = 16L * 1024 * 1024;
+ protected long compactionBatchSize = 16L * 1024 * 1024;
+ protected long compactionInterval = 60L * 1000;
+
+ protected final AtomicLong totalBytes = new AtomicLong(0);
+ protected final BlockingQueue<CompactionFile> fileNames = new PriorityBlockingQueue<>();
+ protected long compactionIndex = 0;
+
+ protected List<IMapFileData> writeBatch = new ArrayList<>();
+
+ protected FileSystem fs;
+ protected Path parentPath;
+ protected Path finalPath;
+ protected Path currentTmpPath;
+ protected Serializer serializer;
+
+ protected volatile boolean isRunning = true;
+
+ protected AbstractLSMWriter(Map<String, Object> config) {
+ if (config.get(FileConstants.FileInitProperties.COMPACTION_THRESHOLD) != null) {
+ long threshold =
+ (long) config.get(FileConstants.FileInitProperties.COMPACTION_THRESHOLD);
+ if (threshold > 0) {
+ compactionThreshold = threshold;
+ }
+ }
+ if (config.get(FileConstants.FileInitProperties.MAX_SINGLE_FILE_SIZE) != null) {
+ long maxSize = (long) config.get(FileConstants.FileInitProperties.MAX_SINGLE_FILE_SIZE);
+ if (maxSize > 0) {
+ maxSingleFileSize = maxSize;
+ }
+ }
+ if (config.get(FileConstants.FileInitProperties.COMPACTION_BATCH_SIZE) != null) {
+ long batchSize =
+ (long) config.get(FileConstants.FileInitProperties.COMPACTION_BATCH_SIZE);
+ if (batchSize > 0 && batchSize >= maxSingleFileSize) {
+ compactionBatchSize = batchSize;
+ } else {
+ throw new IllegalArgumentException(
+ "compaction batch size must be >= max single file size");
+ }
+ }
+ if (config.get(FileConstants.FileInitProperties.COMPACTION_INTERVAL) != null) {
+ long interval = (long) config.get(FileConstants.FileInitProperties.COMPACTION_INTERVAL);
+ if (interval > 0) {
+ compactionInterval = interval;
+ }
+ }
+ }
Review Comment:
How about we convert it this way? That way, even if the user makes a wrong configuration, they can clearly perceive it.
```
private long asLong(Object v, long defaultValue) {
if (v == null) return defaultValue;
if (v instanceof Number) return ((Number) v).longValue();
if (v instanceof String) return Long.parseLong(((String) v).trim());
throw new IllegalArgumentException("Expect number or string, but got: " + v.getClass());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on code in PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#discussion_r2736761916
##########
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/bean/IMapFileData.java:
##########
@@ -27,6 +27,8 @@
import java.io.Serializable;
+import static org.apache.hadoop.hdfs.DFSUtilClient.compareBytes;
Review Comment:
Is there a similar method in the JDK? It's better to use one from the JDK if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-3803686424
@DanielCarter-stack
Issues 1, 7, 8, and 9 have been addressed.
The `write` and `compaction` operations run on separate single threads,
and the only shared resources are `fileNames` and `totalBytes`.
Both of these resources are thread-safe, so we do not expect any concurrency issues.
To be extra cautious, `synchronized` has been added to `recoverFromCrash()` and `write()`.
Regarding the current compaction logic, tombstone records are not deleted
because previous versions of the same keys may exist in other files.
Even if duplicates occur, the compaction process keeps only the latest value
in the merged file, effectively eliminating duplicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Add compaction support to IMAP external storage [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on PR #10399:
URL: https://github.com/apache/seatunnel/pull/10399#issuecomment-4190446913
@DanielLeens Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org