You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "yzeng1618 (via GitHub)" <gi...@apache.org> on 2026/01/13 09:22:23 UTC
[PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
yzeng1618 opened a new pull request, #10332:
URL: https://github.com/apache/seatunnel/pull/10332
…arallel read
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
### Purpose of this pull request
https://github.com/apache/seatunnel/issues/10326
HdfsFile source currently uses “one file = one split”, which limits parallelism when there are only a few huge files.
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
### Does this PR introduce _any_ user-facing change?
yes
1. This PR adds enable_file_split / file_split_size to HdfsFile source and wires HDFS-specific split strategies:
- text/csv/json: split by file_split_size and align split end to the next row_delimiter (HDFS seek-based implementation for large files).
- parquet: split by RowGroup (logical split) and read footer metadata using HadoopConf-backed Configuration (works with Kerberos/HA/NameService).
2. Example:
```
enable_file_split = true
file_split_size = 268435456
row_delimiter = "\n" (for text/csv/json)
```
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
If no, write 'No'.
If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
-->
### How was this patch tested?
1. Unit tests
HdfsFileAccordingToSplitSizeSplitStrategyTest#testReadBySplitsShouldMatchFullRead
3. E2E
HdfsFileIT#testHdfsTextReadWithFileSplit
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
-->
### Check list
* [ * ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ * ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
4. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
5. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
6. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
7. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3754691824
<!-- code-pr-reviewer -->
## Blocking Issues
**BLOCKER: Data loss at split boundaries**
`AccordingToSplitSizeSplitStrategy.findNextDelimiterWithSeek()` (lines 147-165) returns incorrect position when split end aligns with line delimiter. When `endPos >= startPos`, returns `endPos` directly without verifying if it skips the next split's start character, corrupting data in parallel reads.
**CRITICAL: Parquet footer reads fail in Kerberos environments**
`ParquetFileSplitStrategy.getFooter()` (line 133) creates `new Configuration()` instead of using passed `hadoopConf`. Constructor accepts `HadoopConf` parameter (line 67) but `getFooter()` uses empty configuration without Kerberos/HA settings, causing startup failures on Kerberos-enabled HDFS clusters.
**CRITICAL: Missing Kerberos/HDFS HA validation**
No E2E tests cover Kerberos/HA/NameService scenarios (`HdfsFileIT.java`). Tests use `apache/hadoop:3` basic image only; no miniKDC or Kerberos configuration tests, putting production Kerberos deployments at risk.
**CRITICAL: Backward compatibility unverified**
Missing regression tests for default behavior when `enable_file_split` is not configured. Existing jobs may behave unexpectedly after upgrade if default value is incorrect.
**CRITICAL: Missing split boundary edge case tests**
No tests for split boundary at line delimiter, file start/end, or multi-byte character boundaries. Edge cases may cause data duplication or loss; potential for infinite scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2693076721
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileAccordingToSplitSizeSplitStrategy.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplit;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HdfsFileAccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
Review Comment:
Has been revised
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileSplitStrategyFactory.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.connectors.seatunnel.file.config.ArchiveCompressFormat;
+import org.apache.seatunnel.connectors.seatunnel.file.config.CompressFormat;
+import org.apache.seatunnel.connectors.seatunnel.file.config.FileBaseSourceOptions;
+import org.apache.seatunnel.connectors.seatunnel.file.config.FileFormat;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.hdfs.config.HdfsFileHadoopConfig;
+import org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.config.HdfsSourceConfigOptions;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.DefaultFileSplitStrategy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import static org.apache.seatunnel.connectors.seatunnel.file.config.FileBaseSourceOptions.DEFAULT_ROW_DELIMITER;
+
+public class HdfsFileSplitStrategyFactory {
Review Comment:
Has been revised
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsParquetFileSplitStrategy.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplit;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HdfsParquetFileSplitStrategy implements FileSplitStrategy, Closeable {
Review Comment:
Has been revised
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3754886343
<!-- code-pr-reviewer -->
Thanks for implementing this important feature! The overall approach looks solid, but I found 2 CRITICAL issues that should be addressed before merging:
**1. Poor error observability (GEN-002)**
`AccordingToSplitSizeSplitStrategy.java:107-109,115-117` - All IOExceptions are mapped to `FILE_READ_FAILED`, losing distinction between FileNotFoundException, permission denied, network timeout, etc. This makes troubleshooting difficult. Consider catching specific exception types and mapping them to appropriate error codes.
**2. Severe performance regression for large file splits (GEN-003)**
`AbstractReadStrategy.java:510-521` - `safeSlice()` uses `InputStream.skip()` in a loop, which reads byte-by-byte. For HDFS files with non-first splits (e.g., start=10GB), this causes massive network/disk I/O just to reach the offset. HDFS supports `FSDataInputStream.seek()`, which should be used instead. Without this fix, enabling splits on large HDFS files will be slower than not splitting at all.
**Note on COR-001**: The idempotency concern in `addSplitsBack()` couldn't be fully verified from code review. The FileSourceSplit.equals() implementation suggests duplicates would be detected, but runtime behavior should be validated post-merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2688766791
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileAccordingToSplitSizeSplitStrategy.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplit;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HdfsFileAccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
Review Comment:
> Okay, so you still suggest reusing the base code as much as possible? My main concern is that it might affect the logic of the localfile code.
Yes, the sharding logic should be maintained uniformly in the base, and there should not be a lot of code redundancy. It is also convenient to add this function to file system connectors such as S3 and OSS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2692826874
##########
docs/en/connector-v2/source/HdfsFile.md:
##########
@@ -88,6 +88,8 @@ Read data from hdfs file system.
| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. |
| file_filter_modified_start | string | no | - | File modification time filter. The connector will filter some files base on the last modification start time (include start time). The default data format is `yyyy-MM-dd HH:mm:ss`. |
| file_filter_modified_end | string | no | - | File modification time filter. The connector will filter some files base on the last modification end time (not include end time). The default data format is `yyyy-MM-dd HH:mm:ss`. |
+| enable_file_split | boolean | no | false | Turn on logical file split to improve parallelism for huge files. Only supported for `text`/`csv`/`json`/`parquet` and non-compressed format. |
Review Comment:
Do we need this parameter? Can the size parameter be a positive number to indicate that it is enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2703597453
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/ParquetFileSplitStrategy.java:
##########
@@ -78,41 +93,74 @@ List<FileSourceSplit> splitByRowGroups(
return splits;
}
long currentStart = 0;
- long currentLength = 0;
+ long currentEnd = 0;
boolean hasOpenSplit = false;
for (BlockMetaData block : rowGroups) {
long rgStart = block.getStartingPos();
long rgSize = block.getCompressedSize();
+ long rgEnd = rgStart + rgSize;
// start a new split
if (!hasOpenSplit) {
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
hasOpenSplit = true;
continue;
}
// exceeds threshold, close current split
- if (currentLength + rgSize > splitSizeBytes) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (rgEnd - currentStart > splitSizeBytes) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
// start next split
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
} else {
- currentLength += rgSize;
+ currentEnd = rgEnd;
}
}
// last split
- if (hasOpenSplit && currentLength > 0) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (hasOpenSplit && currentEnd > currentStart) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
}
return splits;
}
private List<BlockMetaData> readRowGroups(String filePath) throws IOException {
Path path = new Path(filePath);
- Configuration conf = new Configuration();
- try (ParquetFileReader reader =
- ParquetFileReader.open(HadoopInputFile.fromPath(path, conf))) {
- return reader.getFooter().getBlocks();
+ if (hadoopFileSystemProxy == null) {
Review Comment:
Under what circumstances will `hadoopFileSystemProxy` be null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2688687352
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileAccordingToSplitSizeSplitStrategy.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplit;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HdfsFileAccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
Review Comment:
Okay, so you still suggest reusing the base code as much as possible? My main concern is that it might affect the logic of the localfile code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3794278521
+1 if conflicts solved
<img width="1522" height="338" alt="image" src="https://github.com/user-attachments/assets/a77cf0cd-c8c0-49b2-bf98-f517af718fd2" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2693072645
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/ParquetFileSplitStrategy.java:
##########
@@ -78,41 +93,71 @@ List<FileSourceSplit> splitByRowGroups(
return splits;
}
long currentStart = 0;
- long currentLength = 0;
+ long currentEnd = 0;
boolean hasOpenSplit = false;
for (BlockMetaData block : rowGroups) {
long rgStart = block.getStartingPos();
long rgSize = block.getCompressedSize();
+ long rgEnd = rgStart + rgSize;
// start a new split
if (!hasOpenSplit) {
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
hasOpenSplit = true;
continue;
}
// exceeds threshold, close current split
- if (currentLength + rgSize > splitSizeBytes) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (rgEnd - currentStart > splitSizeBytes) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
// start next split
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
} else {
- currentLength += rgSize;
+ currentEnd = rgEnd;
}
}
// last split
- if (hasOpenSplit && currentLength > 0) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (hasOpenSplit && currentEnd > currentStart) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
}
return splits;
}
private List<BlockMetaData> readRowGroups(String filePath) throws IOException {
Path path = new Path(filePath);
- Configuration conf = new Configuration();
- try (ParquetFileReader reader =
- ParquetFileReader.open(HadoopInputFile.fromPath(path, conf))) {
- return reader.getFooter().getBlocks();
+ if (hadoopFileSystemProxy == null) {
+ Configuration conf = new Configuration();
+ try (ParquetFileReader reader =
+ ParquetFileReader.open(HadoopInputFile.fromPath(path, conf))) {
+ return reader.getFooter().getBlocks();
+ }
+ }
+ try {
+ return hadoopFileSystemProxy.doWithHadoopAuth(
+ (configuration, userGroupInformation) -> {
+ try (ParquetFileReader reader =
+ ParquetFileReader.open(
+ HadoopInputFile.fromPath(path, configuration))) {
+ return reader.getFooter().getBlocks();
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException) {
Review Comment:
Thanks for the feedback. I updated ParquetFileSplitStrategy to rethrow runtime exceptions (so we don’t mask non-IO failures as IOExceptions) and added an HDFS parquet split E2E case to cover the missing parquet scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2692901078
##########
docs/en/connector-v2/source/HdfsFile.md:
##########
@@ -88,6 +88,8 @@ Read data from hdfs file system.
| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](../source-common-options.md) for details. |
| file_filter_modified_start | string | no | - | File modification time filter. The connector will filter some files base on the last modification start time (include start time). The default data format is `yyyy-MM-dd HH:mm:ss`. |
| file_filter_modified_end | string | no | - | File modification time filter. The connector will filter some files base on the last modification end time (not include end time). The default data format is `yyyy-MM-dd HH:mm:ss`. |
+| enable_file_split | boolean | no | false | Turn on logical file split to improve parallelism for huge files. Only supported for `text`/`csv`/`json`/`parquet` and non-compressed format. |
Review Comment:
At present, the parameters are mainly consistent with those of the localfile, and this parameter is used by default to control whether sharding is enabled. https://github.com/apache/seatunnel/pull/10142/changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3794553405
> +1 if conflicts solved <img alt="image" width="1522" height="338" src="https://private-user-images.githubusercontent.com/15833811/540080428-a77cf0cd-c8c0-49b2-bf98-f517af718fd2.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjkyNTc1MzYsIm5iZiI6MTc2OTI1NzIzNiwicGF0aCI6Ii8xNTgzMzgxMS81NDAwODA0MjgtYTc3Y2YwY2QtYzhjMC00OWIyLWJmOTgtZjUxN2FmNzE4ZmQyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAxMjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMTI0VDEyMjAzNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc3NzYyYTVmMzcxNjc3OGIzYTEzOTQ3N2MzNGVkMWY2ODlmMmJiYWI1MmU0YWJkZjgwNjM0YTQ4NGZhYTFiMDEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.MQDTPfuAntB2vsG9ExGFMxrYUSgEaBYQayr8XtfZP8c">
The conflict has been resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3752692678
hi,It seems that a parquetd use case is still missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2692801902
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/ParquetFileSplitStrategy.java:
##########
@@ -78,41 +93,71 @@ List<FileSourceSplit> splitByRowGroups(
return splits;
}
long currentStart = 0;
- long currentLength = 0;
+ long currentEnd = 0;
boolean hasOpenSplit = false;
for (BlockMetaData block : rowGroups) {
long rgStart = block.getStartingPos();
long rgSize = block.getCompressedSize();
+ long rgEnd = rgStart + rgSize;
// start a new split
if (!hasOpenSplit) {
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
hasOpenSplit = true;
continue;
}
// exceeds threshold, close current split
- if (currentLength + rgSize > splitSizeBytes) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (rgEnd - currentStart > splitSizeBytes) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
// start next split
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
} else {
- currentLength += rgSize;
+ currentEnd = rgEnd;
}
}
// last split
- if (hasOpenSplit && currentLength > 0) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (hasOpenSplit && currentEnd > currentStart) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
}
return splits;
}
private List<BlockMetaData> readRowGroups(String filePath) throws IOException {
Path path = new Path(filePath);
- Configuration conf = new Configuration();
- try (ParquetFileReader reader =
- ParquetFileReader.open(HadoopInputFile.fromPath(path, conf))) {
- return reader.getFooter().getBlocks();
+ if (hadoopFileSystemProxy == null) {
+ Configuration conf = new Configuration();
+ try (ParquetFileReader reader =
+ ParquetFileReader.open(HadoopInputFile.fromPath(path, conf))) {
+ return reader.getFooter().getBlocks();
+ }
+ }
+ try {
+ return hadoopFileSystemProxy.doWithHadoopAuth(
+ (configuration, userGroupInformation) -> {
+ try (ParquetFileReader reader =
+ ParquetFileReader.open(
+ HadoopInputFile.fromPath(path, configuration))) {
+ return reader.getFooter().getBlocks();
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException) {
Review Comment:
It seems that capturing accurate exceptions is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3758519598
> Thanks for implementing this important feature! The overall approach looks solid, but I found 2 CRITICAL issues that should be addressed before merging:
>
> **1. Poor error observability (GEN-002)** `AccordingToSplitSizeSplitStrategy.java:107-109,115-117` - All IOExceptions are mapped to `FILE_READ_FAILED`, losing distinction between FileNotFoundException, permission denied, network timeout, etc. This makes troubleshooting difficult. Consider catching specific exception types and mapping them to appropriate error codes.
>
> **2. Severe performance regression for large file splits (GEN-003)** `AbstractReadStrategy.java:510-521` - `safeSlice()` uses `InputStream.skip()` in a loop, which reads byte-by-byte. For HDFS files with non-first splits (e.g., start=10GB), this causes massive network/disk I/O just to reach the offset. HDFS supports `FSDataInputStream.seek()`, which should be used instead. Without this fix, enabling splits on large HDFS files will be slower than not splitting at all.
>
> **Note on COR-001**: The idempotency concern in `addSplitsBack()` couldn't be fully verified from code review. The FileSourceSplit.equals() implementation suggests duplicates would be detected, but runtime behavior should be validated post-merge.
The repair has been completed as required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2685746872
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileAccordingToSplitSizeSplitStrategy.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplit;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HdfsFileAccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
Review Comment:
```suggestion
public class HdfsFileAccordingToSplitSizeSplitStrategy extends AccordingToSplitSizeSplitStrategy{
```
1、Modify `AccordingToSplitSizeSplitStrategy` to introduce `HadoopFileSystemProxy`
2、Delete `LocalFileAccordingToSplitSizeSplitStrategy`
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileSplitStrategyFactory.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.connectors.seatunnel.file.config.ArchiveCompressFormat;
+import org.apache.seatunnel.connectors.seatunnel.file.config.CompressFormat;
+import org.apache.seatunnel.connectors.seatunnel.file.config.FileBaseSourceOptions;
+import org.apache.seatunnel.connectors.seatunnel.file.config.FileFormat;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.hdfs.config.HdfsFileHadoopConfig;
+import org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.config.HdfsSourceConfigOptions;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.DefaultFileSplitStrategy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import static org.apache.seatunnel.connectors.seatunnel.file.config.FileBaseSourceOptions.DEFAULT_ROW_DELIMITER;
+
+public class HdfsFileSplitStrategyFactory {
Review Comment:
```suggestion
public class FileSplitStrategyFactory {
```
Delete `LocalFileSplitStrategyFactory`
##########
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsParquetFileSplitStrategy.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.split;
+
+import org.apache.seatunnel.common.exception.SeaTunnelRuntimeException;
+import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
+import org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorErrorCode;
+import org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSourceSplit;
+import org.apache.seatunnel.connectors.seatunnel.file.source.split.FileSplitStrategy;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.metadata.BlockMetaData;
+import org.apache.parquet.hadoop.util.HadoopInputFile;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class HdfsParquetFileSplitStrategy implements FileSplitStrategy, Closeable {
Review Comment:
Remove `HdfsParquetFileSplitStrategy` and enhance `ParquetFileSplitStrategy`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on code in PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#discussion_r2704160353
##########
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/ParquetFileSplitStrategy.java:
##########
@@ -78,41 +93,74 @@ List<FileSourceSplit> splitByRowGroups(
return splits;
}
long currentStart = 0;
- long currentLength = 0;
+ long currentEnd = 0;
boolean hasOpenSplit = false;
for (BlockMetaData block : rowGroups) {
long rgStart = block.getStartingPos();
long rgSize = block.getCompressedSize();
+ long rgEnd = rgStart + rgSize;
// start a new split
if (!hasOpenSplit) {
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
hasOpenSplit = true;
continue;
}
// exceeds threshold, close current split
- if (currentLength + rgSize > splitSizeBytes) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (rgEnd - currentStart > splitSizeBytes) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
// start next split
currentStart = rgStart;
- currentLength = rgSize;
+ currentEnd = rgEnd;
} else {
- currentLength += rgSize;
+ currentEnd = rgEnd;
}
}
// last split
- if (hasOpenSplit && currentLength > 0) {
- splits.add(new FileSourceSplit(tableId, filePath, currentStart, currentLength));
+ if (hasOpenSplit && currentEnd > currentStart) {
+ splits.add(
+ new FileSourceSplit(
+ tableId, filePath, currentStart, currentEnd - currentStart));
}
return splits;
}
private List<BlockMetaData> readRowGroups(String filePath) throws IOException {
Path path = new Path(filePath);
- Configuration conf = new Configuration();
- try (ParquetFileReader reader =
- ParquetFileReader.open(HadoopInputFile.fromPath(path, conf))) {
- return reader.getFooter().getBlocks();
+ if (hadoopFileSystemProxy == null) {
Review Comment:
hadoopFileSystemProxy is only null when ParquetFileSplitStrategy is instantiated via the constructor without HadoopConf (mainly for unit tests / direct usage). In the normal connector path when file split is enabled, FileSplitStrategyFactory requires a non-null HadoopConf and always creates ParquetFileSplitStrategy with it, so the proxy won’t be null at runtime. The null branch just falls back to using a default Configuration without any Hadoop auth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3799270958
> ### Issue 1: Binary Compatibility Break - Removal of LocalFileAccordingToSplitSizeSplitStrategy
> **Location**:
>
> * `seatunnel-connectors-v2/connector-file/connector-file-local/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/local/source/split/LocalFileAccordingToSplitSizeSplitStrategy.java` (deleted)
>
> **Related Context**:
>
> * Original class: `LocalFileAccordingToSplitSizeSplitStrategy` extends `AccordingToSplitSizeSplitStrategy`
> * Replaced by: Direct use of `AccordingToSplitSizeSplitStrategy` + `LocalFileHadoopConfig`
>
> **Problem Description**: The PR deleted the `LocalFileAccordingToSplitSizeSplitStrategy` class. If external users or third-party Connectors inherit from this class, compilation errors or runtime `ClassNotFoundException` will occur after upgrade.
>
> **Potential Risks**:
>
> * **Risk 1**: External inheritors will fail to compile (if source dependency)
> * **Risk 2**: Third-party Connectors compiled with old versions will have runtime `ClassNotFoundException` (if binary dependency)
> * **Risk 3**: Although File Connector is an internal module, there is no guarantee that users haven't extended based on it
>
> **Impact Scope**:
>
> * **Direct Impact**: None (no inheritance within SeaTunnel)
> * **Indirect Impact**: Possible third-party Connectors
> * **Affected Area**: LocalFile Connector
>
> **Severity**: MINOR
>
> **Improvement Suggestions**:
>
> ```java
> // Keep old class as deprecated adapter
> @Deprecated
> public class LocalFileAccordingToSplitSizeSplitStrategy extends AccordingToSplitSizeSplitStrategy {
> public LocalFileAccordingToSplitSizeSplitStrategy(
> String rowDelimiter, long skipHeaderRowNumber, String encodingName, long splitSize) {
> super(new LocalFileHadoopConfig(), rowDelimiter, skipHeaderRowNumber, encodingName, splitSize);
> }
> }
> ```
>
> **Rationale**: Keep one version as a deprecated adapter to give users migration time, and completely remove it in version 2.8.0.
>
> ### Issue 2: Boundary Condition Defect - delimiterBytes.length == 0 causes seek failure
> **Location**: `AccordingToSplitSizeSplitStrategy.java:207-209`
>
> ```java
> long scanStart = Math.max(0, startPos - (delimiterBytes.length - 1));
> input.seek(scanStart);
> ```
>
> **Related Context**:
>
> * Constructor validation: `if (rowDelimiter == null || rowDelimiter.isEmpty())` guards against null, but not against length 0
> * Actually `"".isEmpty() == true`, so the constructor will intercept, but this is an implicit dependency
>
> **Problem Description**: Although the constructor will intercept `rowDelimiter.isEmpty()`, the algorithm in `findNextDelimiterWithSeek` assumes `delimiterBytes.length >= 1`. If someone modifies constructor validation logic in the future (e.g., allowing empty delimiter to indicate no splitting), a bug will occur here.
>
> A more serious problem is: when `delimiterBytes.length == 1`, `scanStart = startPos - 0 = startPos`, the algorithm is correct; but when future support for multi-byte delimiter is added (already supported), the algorithm assumes `scanStart` might be less than `startPos` to capture cross-boundary delimiters.
>
> **Potential Risks**:
>
> * **Risk 1**: If constructor validation is bypassed (e.g., reflection, deserialization attack), `delimiterBytes.length == 0` will cause `scanStart = startPos - (-1) = startPos + 1`, seek exceeds file range
> * **Risk 2**: Code maintainers may not understand why `length - 1`, mistakenly changing it to other values
>
> **Impact Scope**:
>
> * **Direct Impact**: `AccordingToSplitSizeSplitStrategy.findNextDelimiterWithSeek()`
> * **Indirect Impact**: All scenarios using text/csv/json splitting
> * **Affected Area**: HdfsFile, LocalFile
>
> **Severity**: MINOR
>
> **Improvement Suggestions**:
>
> ```java
> private long findNextDelimiterWithSeek(FSDataInputStream input, long startPos, long fileSize)
> throws IOException {
> // Explicit assertion to clarify algorithm assumptions
> if (delimiterBytes.length == 0) {
> throw new IllegalStateException("delimiterBytes must not be empty");
> }
>
> // Extract magic numbers as constants
> long scanStart = Math.max(0, startPos - (delimiterBytes.length - 1));
> input.seek(scanStart);
> // ...
> }
> ```
>
> **Rationale**: Add defensive assertions to make algorithm assumptions explicit even though the constructor has validated, facilitating future maintenance.
>
> ### Issue 3: API Compatibility - AccordingToSplitSizeSplitStrategy changed from abstract to concrete
> **Location**: `AccordingToSplitSizeSplitStrategy.java:50`
>
> ```java
> // Old version
> public abstract class AccordingToSplitSizeSplitStrategy implements FileSplitStrategy {
> protected abstract InputStream getInputStream(String filePath) throws IOException;
> protected abstract long getFileSize(String filePath) throws IOException;
> }
>
> // New version
> public class AccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
> public AccordingToSplitSizeSplitStrategy(HadoopConf hadoopConf, ...) {
> this.hadoopFileSystemProxy = new HadoopFileSystemProxy(hadoopConf);
> }
> }
> ```
>
> **Related Context**:
>
> * Subclass: `LocalFileAccordingToSplitSizeSplitStrategy` (deleted)
> * No other known subclasses
>
> **Problem Description**: `AccordingToSplitSizeSplitStrategy` changed from abstract class to concrete class, and constructor signature changed from `(String, long, String, long)` to `(HadoopConf, String, long, String, long)`. This is a breaking change.
>
> Although there are no other subclasses within SeaTunnel, if external extensions inherit from this class, they will:
>
> 1. Compilation error: unable to instantiate parent class
> 2. Abstract methods `getInputStream()`/`getFileSize()` disappear, subclass compilation fails
>
> **Potential Risks**:
>
> * **Risk 1**: Third-party Connectors inheriting this class will fail to compile
> * **Risk 2**: In a plugin architecture, user-defined File Sources may be affected
>
> **Impact Scope**:
>
> * **Direct Impact**: All classes inheriting `AccordingToSplitSizeSplitStrategy` (none internally)
> * **Indirect Impact**: Possible third-party extensions
> * **Affected Area**: File Connector ecosystem
>
> **Severity**: MINOR
>
> **Improvement Suggestions**: Option 1 (Recommended): Explicitly mark Breaking Changes in release notes, requiring users upgrading to 2.7.x to check if they inherit this class.
>
> Option 2: Keep abstract base class, add concrete implementation
>
> ```java
> // Keep old abstract class (deprecated)
> @Deprecated
> public abstract class LegacyAccordingToSplitSizeSplitStrategy implements FileSplitStrategy {
> // Old abstract method
> protected abstract InputStream getInputStream(String filePath) throws IOException;
> protected abstract long getFileSize(String filePath) throws IOException;
> }
>
> // New concrete implementation
> public class AccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
> // New implementation
> }
> ```
>
> **Rationale**: Although there are no internal subclasses currently, as a framework library, API stability should be maintained, at least marking `@Deprecated` to provide a transition period.
>
> ### Issue 4: Code Duplication - HdfsFileSource and LocalFileSource have identical initialization logic
> **Location**:
>
> * `HdfsFileSource.java:36-48` (newly added)
> * `LocalFileSource.java:35-47` (modified)
>
> **Related Context**:
>
> ```java
> // HdfsFileSource.java
> private static FileSplitStrategy initFileSplitStrategy(MultipleTableHdfsFileSourceConfig config) {
> Map<String, FileSplitStrategy> splitStrategies = new HashMap<>();
> for (BaseFileSourceConfig fileSourceConfig : config.getFileSourceConfigs()) {
> String tableId = fileSourceConfig.getCatalogTable().getTableId().toTablePath().toString();
> splitStrategies.put(tableId,
> FileSplitStrategyFactory.initFileSplitStrategy(
> fileSourceConfig.getBaseFileSourceConfig(),
> fileSourceConfig.getHadoopConfig()));
> }
> return new MultipleTableFileSplitStrategy(splitStrategies);
> }
>
> // LocalFileSource.java - Exactly the same!
> private static FileSplitStrategy initFileSplitStrategy(MultipleTableLocalFileSourceConfig sourceConfig) {
> Map<String, FileSplitStrategy> splitStrategies = new HashMap<>();
> for (BaseFileSourceConfig fileSourceConfig : sourceConfig.getFileSourceConfigs()) {
> String tableId = fileSourceConfig.getCatalogTable().getTableId().toTablePath().toString();
> splitStrategies.put(tableId,
> FileSplitStrategyFactory.initFileSplitStrategy(
> fileSourceConfig.getBaseFileSourceConfig(),
> fileSourceConfig.getHadoopConfig()));
> }
> return new MultipleTableFileSplitStrategy(splitStrategies);
> }
> ```
>
> **Problem Description**: The `initFileSplitStrategy()` method logic in both classes is completely identical, violating the DRY (Don't Repeat Yourself) principle. If initialization logic needs to be modified in the future (e.g., adding caching, logging), two places need to be modified simultaneously.
>
> **Potential Risks**:
>
> * **Risk 1**: Increased maintenance cost (synchronized modifications to two places)
> * **Risk 2**: Easy to create inconsistencies (e.g., modifying only one place, forgetting the other)
>
> **Impact Scope**:
>
> * **Direct Impact**: `HdfsFileSource`, `LocalFileSource`
> * **Indirect Impact**: Newly added FileSources in the future (e.g., S3FileSource)
> * **Affected Area**: All File Sources
>
> **Severity**: MINOR
>
> **Improvement Suggestions**:
>
> ```java
> // BaseMultipleTableFileSource.java
> public abstract class BaseMultipleTableFileSource {
> protected static FileSplitStrategy initFileSplitStrategy(
> BaseMultipleTableFileSourceConfig sourceConfig) {
> Map<String, FileSplitStrategy> splitStrategies = new HashMap<>();
> for (BaseFileSourceConfig fileSourceConfig : sourceConfig.getFileSourceConfigs()) {
> String tableId = fileSourceConfig.getCatalogTable().getTableId().toTablePath().toString();
> splitStrategies.put(tableId,
> FileSplitStrategyFactory.initFileSplitStrategy(
> fileSourceConfig.getBaseFileSourceConfig(),
> fileSourceConfig.getHadoopConfig()));
> }
> return new MultipleTableFileSplitStrategy(splitStrategies);
> }
> }
>
> // HdfsFileSource.java
> private HdfsFileSource(MultipleTableHdfsFileSourceConfig sourceConfig) {
> super(sourceConfig, initFileSplitStrategy(sourceConfig)); // Call parent class method
> }
>
> // LocalFileSource.java
> private LocalFileSource(MultipleTableLocalFileSourceConfig sourceConfig) {
> super(sourceConfig, initFileSplitStrategy(sourceConfig)); // Call parent class method
> }
> ```
>
> **Rationale**: Extract common logic to parent class to reduce duplication and improve maintainability.
>
> ### Issue 6: Incomplete Documentation - Missing performance recommendations and limitation notes
> **Location**: `docs/en/connectors/source/HdfsFile.md:256-265`
>
> **Related Context**: Current documentation:
>
> ```
> ### enable_file_split [boolean]
> Turn on the file splitting function, the default is false. ...
>
> ### file_split_size [long]
> File split size, which can be filled in when the enable_file_split parameter is true. ...
> The default value is the number of bytes of 128MB, which is 134217728.
> ```
>
> **Problem Description**: The documentation is missing the following key information:
>
> 1. **Performance Recommendations**: In which scenarios should it be enabled? In which scenarios should it not be?
> 2. **Limitation Notes**: Maximum split number limit? Memory overhead?
> 3. **Tuning Guide**: How to choose an appropriate `file_split_size`?
> 4. **Known Issues**: For example, compressed files, archive files do not support splitting
>
> **Potential Risks**:
>
> * **Risk 1**: Users abuse the feature (e.g., enabling splitting for small files) resulting in performance degradation
> * **Risk 2**: Users set too small split_size (e.g., 1KB) causing OOM
> * **Risk 3**: Users expect compressed files to also be splittable, actually not supported
>
> **Impact Scope**:
>
> * **Direct Impact**: User experience
> * **Indirect Impact**: Increased support cost
> * **Affected Area**: HdfsFile, LocalFile
>
> **Severity**: MINOR
>
> **Improvement Suggestions**:
>
> ```
> ### enable_file_split [boolean]
> Turn on the file splitting function, the default is false. ...
>
> **Performance Recommendations:**
> - **Enable when**: Reading few large files (> 1GB each) with parallelism > 4
> - **Disable when**: Reading many small files (< 128MB each) or parallelism = 1
> - **Not supported for**: Compressed files (gzip, bzip2), archive files (zip, tar)
>
> **Limitations:**
> - Maximum splits per file: 100,000 (configurable via `seaunnel.source.file.split.max-count`)
> - Memory overhead: ~100 bytes per split
>
> ### file_split_size [long]
> File split size, which can be filled in when the enable_file_split parameter is true. ...
>
> **Tuning Guidelines:**
> - **Recommended**: 128MB - 512MB for HDFS, 64MB - 256MB for local file
> - **Too small (< 16MB)**: Excessive splits, high overhead
> - **Too large (> 1GB)**: Insufficient parallelism
> - **Formula**: `file_split_size = total_file_size / desired_parallelism`
> ```
>
> **Rationale**: Add performance guidelines and limitation notes to help users use the feature correctly.
>
> ### Issue 7: Architectural Complexity - LocalFile requiring HadoopConf configuration concept overflow
> **Location**: `LocalFileSource.java:39-42`
>
> ```java
> splitStrategies.put(tableId,
> FileSplitStrategyFactory.initFileSplitStrategy(
> fileSourceConfig.getBaseFileSourceConfig(),
> fileSourceConfig.getHadoopConfig())); // ← Does LocalFile need HadoopConf?
> ```
>
> **Related Context**:
>
> * `LocalFileHadoopConfig`: LocalFile's HadoopConf implementation (actually just configuring `file:///`)
> * `AccordingToSplitSizeSplitStrategy`: requires `HadoopConf` to create `HadoopFileSystemProxy`
>
> **Problem Description**: To reuse `AccordingToSplitSizeSplitStrategy`, LocalFile scenarios are also forced to introduce the `HadoopConf` concept. Although technically feasible (through `LocalFileHadoopConfig`), it is conceptually unnatural:
>
> * LocalFile users typically don't care about Hadoop configuration
> * Error messages may contain "Hadoop" wording, confusing LocalFile users
>
> **Potential Risks**:
>
> * **Risk 1**: User confusion ("Why do I need Hadoop configuration to read local files?")
> * **Risk 2**: Misleading error messages (e.g., "Hadoop configuration error")
>
> **Impact Scope**:
>
> * **Direct Impact**: LocalFile Source user experience
> * **Indirect Impact**: Documentation complexity
> * **Affected Area**: LocalFile Connector
>
> **Severity**: MINOR
>
> **Improvement Suggestions**: Option 1 (Recommended): Explicitly state in documentation
>
> ```
> For **LocalFile**, the `hadoop_conf` is optional and defaults to `file:///` schema.
> You rarely need to configure it unless you have custom Hadoop settings.
> ```
>
> Option 2 (Complex): Abstract `FileSystemProxy` interface
>
> ```java
> public interface FileSystemProxy {
> FSDataInputStream getInputStream(String path) throws IOException;
> FileStatus getFileStatus(String path) throws IOException;
> }
>
> public class HadoopFileSystemProxy implements FileSystemProxy { ... }
> public class LocalFileSystemProxy implements FileSystemProxy { ... } // Pure Java NIO
> ```
>
> **Rationale**: Option 1 is simple and effective, Option 2 has clearer architecture but requires more work. The current solution is acceptable.
>
> ### Issue 8: Checkpoint Compatibility Not Fully Validated
> **Location**: `FileSourceSplit.java:44-49`
>
> ```java
> public FileSourceSplit(String tableId, String filePath, long start, long length) {
> this.tableId = tableId;
> this.filePath = filePath;
> this.start = start; // ← New field assigned in constructor
> this.length = length; // ← New field assigned in constructor
> }
> ```
>
> **Related Context**:
>
> * `serialVersionUID = 1L`
> * Old version `FileSourceSplit` may only have `tableId`/`filePath`, `start`/`length` use default values 0/-1
>
> **Problem Description**: The PR did not validate the behavior of restoring from old version (without file split feature) checkpoints to the new version:
>
> 1. After deserializing `FileSourceSplit` from old checkpoint, `start=0, length=-1`
> 2. New code will read by byte range, `length=-1` may cause problems
> 3. Need to check if `AbstractReadStrategy.actualRead()` correctly handles `length=-1`
>
> **Validation**: Check `AbstractReadStrategy.safeSlice()`:
>
> ```java
> protected static InputStream safeSlice(InputStream in, long start, long length) throws IOException {
> // ...
> return new BoundedInputStream(in, length); // How does BoundedInputStream handle length=-1?
> }
> ```
>
> Apache Commons IO's `BoundedInputStream(-1)` will read to the end of the stream, so the behavior is correct. But this is an implicit dependency.
>
> **Potential Risks**:
>
> * **Risk 1**: Jobs upgrading from 2.6.x to 2.7.x, after checkpoint recovery, data inconsistency
> * **Risk 2**: The semantics of `length=-1` are not clear enough (does it mean "read to end of file" or "disable split"?)
>
> **Impact Scope**:
>
> * **Direct Impact**: Job recovery for upgrading users
> * **Indirect Impact**: Data consistency
> * **Affected Area**: All jobs using File Source
>
> **Severity**: MAJOR
>
> **Improvement Suggestions**:
>
> ```java
> // FileSourceSplit.java
> public FileSourceSplit(String tableId, String filePath, long start, long length) {
> this.tableId = tableId;
> this.filePath = filePath;
> this.start = start;
> // Clarify semantics: -1 means "entire file" (compatible with old version)
> this.length = length;
> }
>
> // AbstractReadStrategy.java
> protected static InputStream safeSlice(InputStream in, long start, long length) throws IOException {
> if (start > 0) {
> if (in instanceof Seekable) {
> ((Seekable) in).seek(start);
> } else {
> // ...
> }
> }
> // length=-1 means unlimited (compatible with old checkpoint)
> if (length < 0) {
> return in;
> }
> return new BoundedInputStream(in, length);
> }
> ```
>
> Also add tests:
>
> ```java
> @Test
> void testDeserializeFromOldCheckpoint() {
> // Simulate FileSourceSplit from old version serialization
> FileSourceSplit oldSplit = new FileSourceSplit("table", "path"); // start=0, length=-1
> // Verify reading entire file
> List<String> records = readBySplit(oldSplit);
> Assertions.assertEquals(fullFileRecords, records);
> }
> ```
>
> **Rationale**: Clarify the semantics of `length=-1`, add tests to verify upgrade compatibility.
The above suggestions have been implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Connector-V2][HdfsFile] Support true large-file split for parallel read [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on PR #10332:
URL: https://github.com/apache/seatunnel/pull/10332#issuecomment-3794753345
<!-- code-pr-reviewer -->
<!-- cpr:pr_reply_v2_parts {"group": "apache/seatunnel#10332", "part": 1, "total": 1} -->
### Issue 1: Binary Compatibility Break - Removal of LocalFileAccordingToSplitSizeSplitStrategy
**Location**:
- `seatunnel-connectors-v2/connector-file/connector-file-local/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/local/source/split/LocalFileAccordingToSplitSizeSplitStrategy.java` (deleted)
**Related Context**:
- Original class: `LocalFileAccordingToSplitSizeSplitStrategy` extends `AccordingToSplitSizeSplitStrategy`
- Replaced by: Direct use of `AccordingToSplitSizeSplitStrategy` + `LocalFileHadoopConfig`
**Problem Description**:
The PR deleted the `LocalFileAccordingToSplitSizeSplitStrategy` class. If external users or third-party Connectors inherit from this class, compilation errors or runtime `ClassNotFoundException` will occur after upgrade.
**Potential Risks**:
- **Risk 1**: External inheritors will fail to compile (if source dependency)
- **Risk 2**: Third-party Connectors compiled with old versions will have runtime `ClassNotFoundException` (if binary dependency)
- **Risk 3**: Although File Connector is an internal module, there is no guarantee that users haven't extended based on it
**Impact Scope**:
- **Direct Impact**: None (no inheritance within SeaTunnel)
- **Indirect Impact**: Possible third-party Connectors
- **Affected Area**: LocalFile Connector
**Severity**: MINOR
**Improvement Suggestions**:
```java
// Keep old class as deprecated adapter
@Deprecated
public class LocalFileAccordingToSplitSizeSplitStrategy extends AccordingToSplitSizeSplitStrategy {
public LocalFileAccordingToSplitSizeSplitStrategy(
String rowDelimiter, long skipHeaderRowNumber, String encodingName, long splitSize) {
super(new LocalFileHadoopConfig(), rowDelimiter, skipHeaderRowNumber, encodingName, splitSize);
}
}
```
**Rationale**: Keep one version as a deprecated adapter to give users migration time, and completely remove it in version 2.8.0.
---
### Issue 2: Boundary Condition Defect - delimiterBytes.length == 0 causes seek failure
**Location**:
`AccordingToSplitSizeSplitStrategy.java:207-209`
```java
long scanStart = Math.max(0, startPos - (delimiterBytes.length - 1));
input.seek(scanStart);
```
**Related Context**:
- Constructor validation: `if (rowDelimiter == null || rowDelimiter.isEmpty())` guards against null, but not against length 0
- Actually `"".isEmpty() == true`, so the constructor will intercept, but this is an implicit dependency
**Problem Description**:
Although the constructor will intercept `rowDelimiter.isEmpty()`, the algorithm in `findNextDelimiterWithSeek` assumes `delimiterBytes.length >= 1`. If someone modifies constructor validation logic in the future (e.g., allowing empty delimiter to indicate no splitting), a bug will occur here.
A more serious problem is: when `delimiterBytes.length == 1`, `scanStart = startPos - 0 = startPos`, the algorithm is correct; but when future support for multi-byte delimiter is added (already supported), the algorithm assumes `scanStart` might be less than `startPos` to capture cross-boundary delimiters.
**Potential Risks**:
- **Risk 1**: If constructor validation is bypassed (e.g., reflection, deserialization attack), `delimiterBytes.length == 0` will cause `scanStart = startPos - (-1) = startPos + 1`, seek exceeds file range
- **Risk 2**: Code maintainers may not understand why `length - 1`, mistakenly changing it to other values
**Impact Scope**:
- **Direct Impact**: `AccordingToSplitSizeSplitStrategy.findNextDelimiterWithSeek()`
- **Indirect Impact**: All scenarios using text/csv/json splitting
- **Affected Area**: HdfsFile, LocalFile
**Severity**: MINOR
**Improvement Suggestions**:
```java
private long findNextDelimiterWithSeek(FSDataInputStream input, long startPos, long fileSize)
throws IOException {
// Explicit assertion to clarify algorithm assumptions
if (delimiterBytes.length == 0) {
throw new IllegalStateException("delimiterBytes must not be empty");
}
// Extract magic numbers as constants
long scanStart = Math.max(0, startPos - (delimiterBytes.length - 1));
input.seek(scanStart);
// ...
}
```
**Rationale**: Add defensive assertions to make algorithm assumptions explicit even though the constructor has validated, facilitating future maintenance.
---
### Issue 3: API Compatibility - AccordingToSplitSizeSplitStrategy changed from abstract to concrete
**Location**:
`AccordingToSplitSizeSplitStrategy.java:50`
```java
// Old version
public abstract class AccordingToSplitSizeSplitStrategy implements FileSplitStrategy {
protected abstract InputStream getInputStream(String filePath) throws IOException;
protected abstract long getFileSize(String filePath) throws IOException;
}
// New version
public class AccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
public AccordingToSplitSizeSplitStrategy(HadoopConf hadoopConf, ...) {
this.hadoopFileSystemProxy = new HadoopFileSystemProxy(hadoopConf);
}
}
```
**Related Context**:
- Subclass: `LocalFileAccordingToSplitSizeSplitStrategy` (deleted)
- No other known subclasses
**Problem Description**:
`AccordingToSplitSizeSplitStrategy` changed from abstract class to concrete class, and constructor signature changed from `(String, long, String, long)` to `(HadoopConf, String, long, String, long)`. This is a breaking change.
Although there are no other subclasses within SeaTunnel, if external extensions inherit from this class, they will:
1. Compilation error: unable to instantiate parent class
2. Abstract methods `getInputStream()`/`getFileSize()` disappear, subclass compilation fails
**Potential Risks**:
- **Risk 1**: Third-party Connectors inheriting this class will fail to compile
- **Risk 2**: In a plugin architecture, user-defined File Sources may be affected
**Impact Scope**:
- **Direct Impact**: All classes inheriting `AccordingToSplitSizeSplitStrategy` (none internally)
- **Indirect Impact**: Possible third-party extensions
- **Affected Area**: File Connector ecosystem
**Severity**: MINOR
**Improvement Suggestions**:
Option 1 (Recommended): Explicitly mark Breaking Changes in release notes, requiring users upgrading to 2.7.x to check if they inherit this class.
Option 2: Keep abstract base class, add concrete implementation
```java
// Keep old abstract class (deprecated)
@Deprecated
public abstract class LegacyAccordingToSplitSizeSplitStrategy implements FileSplitStrategy {
// Old abstract method
protected abstract InputStream getInputStream(String filePath) throws IOException;
protected abstract long getFileSize(String filePath) throws IOException;
}
// New concrete implementation
public class AccordingToSplitSizeSplitStrategy implements FileSplitStrategy, Closeable {
// New implementation
}
```
**Rationale**: Although there are no internal subclasses currently, as a framework library, API stability should be maintained, at least marking `@Deprecated` to provide a transition period.
---
### Issue 4: Code Duplication - HdfsFileSource and LocalFileSource have identical initialization logic
**Location**:
- `HdfsFileSource.java:36-48` (newly added)
- `LocalFileSource.java:35-47` (modified)
**Related Context**:
```java
// HdfsFileSource.java
private static FileSplitStrategy initFileSplitStrategy(MultipleTableHdfsFileSourceConfig config) {
Map<String, FileSplitStrategy> splitStrategies = new HashMap<>();
for (BaseFileSourceConfig fileSourceConfig : config.getFileSourceConfigs()) {
String tableId = fileSourceConfig.getCatalogTable().getTableId().toTablePath().toString();
splitStrategies.put(tableId,
FileSplitStrategyFactory.initFileSplitStrategy(
fileSourceConfig.getBaseFileSourceConfig(),
fileSourceConfig.getHadoopConfig()));
}
return new MultipleTableFileSplitStrategy(splitStrategies);
}
// LocalFileSource.java - Exactly the same!
private static FileSplitStrategy initFileSplitStrategy(MultipleTableLocalFileSourceConfig sourceConfig) {
Map<String, FileSplitStrategy> splitStrategies = new HashMap<>();
for (BaseFileSourceConfig fileSourceConfig : sourceConfig.getFileSourceConfigs()) {
String tableId = fileSourceConfig.getCatalogTable().getTableId().toTablePath().toString();
splitStrategies.put(tableId,
FileSplitStrategyFactory.initFileSplitStrategy(
fileSourceConfig.getBaseFileSourceConfig(),
fileSourceConfig.getHadoopConfig()));
}
return new MultipleTableFileSplitStrategy(splitStrategies);
}
```
**Problem Description**:
The `initFileSplitStrategy()` method logic in both classes is completely identical, violating the DRY (Don't Repeat Yourself) principle. If initialization logic needs to be modified in the future (e.g., adding caching, logging), two places need to be modified simultaneously.
**Potential Risks**:
- **Risk 1**: Increased maintenance cost (synchronized modifications to two places)
- **Risk 2**: Easy to create inconsistencies (e.g., modifying only one place, forgetting the other)
**Impact Scope**:
- **Direct Impact**: `HdfsFileSource`, `LocalFileSource`
- **Indirect Impact**: Newly added FileSources in the future (e.g., S3FileSource)
- **Affected Area**: All File Sources
**Severity**: MINOR
**Improvement Suggestions**:
```java
// BaseMultipleTableFileSource.java
public abstract class BaseMultipleTableFileSource {
protected static FileSplitStrategy initFileSplitStrategy(
BaseMultipleTableFileSourceConfig sourceConfig) {
Map<String, FileSplitStrategy> splitStrategies = new HashMap<>();
for (BaseFileSourceConfig fileSourceConfig : sourceConfig.getFileSourceConfigs()) {
String tableId = fileSourceConfig.getCatalogTable().getTableId().toTablePath().toString();
splitStrategies.put(tableId,
FileSplitStrategyFactory.initFileSplitStrategy(
fileSourceConfig.getBaseFileSourceConfig(),
fileSourceConfig.getHadoopConfig()));
}
return new MultipleTableFileSplitStrategy(splitStrategies);
}
}
// HdfsFileSource.java
private HdfsFileSource(MultipleTableHdfsFileSourceConfig sourceConfig) {
super(sourceConfig, initFileSplitStrategy(sourceConfig)); // Call parent class method
}
// LocalFileSource.java
private LocalFileSource(MultipleTableLocalFileSourceConfig sourceConfig) {
super(sourceConfig, initFileSplitStrategy(sourceConfig)); // Call parent class method
}
```
**Rationale**: Extract common logic to parent class to reduce duplication and improve maintainability.
---
### Issue 6: Incomplete Documentation - Missing performance recommendations and limitation notes
**Location**:
`docs/en/connectors/source/HdfsFile.md:256-265`
**Related Context**:
Current documentation:
```
### enable_file_split [boolean]
Turn on the file splitting function, the default is false. ...
### file_split_size [long]
File split size, which can be filled in when the enable_file_split parameter is true. ...
The default value is the number of bytes of 128MB, which is 134217728.
```
**Problem Description**:
The documentation is missing the following key information:
1. **Performance Recommendations**: In which scenarios should it be enabled? In which scenarios should it not be?
2. **Limitation Notes**: Maximum split number limit? Memory overhead?
3. **Tuning Guide**: How to choose an appropriate `file_split_size`?
4. **Known Issues**: For example, compressed files, archive files do not support splitting
**Potential Risks**:
- **Risk 1**: Users abuse the feature (e.g., enabling splitting for small files) resulting in performance degradation
- **Risk 2**: Users set too small split_size (e.g., 1KB) causing OOM
- **Risk 3**: Users expect compressed files to also be splittable, actually not supported
**Impact Scope**:
- **Direct Impact**: User experience
- **Indirect Impact**: Increased support cost
- **Affected Area**: HdfsFile, LocalFile
**Severity**: MINOR
**Improvement Suggestions**:
```markdown
### enable_file_split [boolean]
Turn on the file splitting function, the default is false. ...
**Performance Recommendations:**
- **Enable when**: Reading few large files (> 1GB each) with parallelism > 4
- **Disable when**: Reading many small files (< 128MB each) or parallelism = 1
- **Not supported for**: Compressed files (gzip, bzip2), archive files (zip, tar)
**Limitations:**
- Maximum splits per file: 100,000 (configurable via `seaunnel.source.file.split.max-count`)
- Memory overhead: ~100 bytes per split
### file_split_size [long]
File split size, which can be filled in when the enable_file_split parameter is true. ...
**Tuning Guidelines:**
- **Recommended**: 128MB - 512MB for HDFS, 64MB - 256MB for local file
- **Too small (< 16MB)**: Excessive splits, high overhead
- **Too large (> 1GB)**: Insufficient parallelism
- **Formula**: `file_split_size = total_file_size / desired_parallelism`
```
**Rationale**: Add performance guidelines and limitation notes to help users use the feature correctly.
---
### Issue 7: Architectural Complexity - LocalFile requiring HadoopConf configuration concept overflow
**Location**:
`LocalFileSource.java:39-42`
```java
splitStrategies.put(tableId,
FileSplitStrategyFactory.initFileSplitStrategy(
fileSourceConfig.getBaseFileSourceConfig(),
fileSourceConfig.getHadoopConfig())); // ← Does LocalFile need HadoopConf?
```
**Related Context**:
- `LocalFileHadoopConfig`: LocalFile's HadoopConf implementation (actually just configuring `file:///`)
- `AccordingToSplitSizeSplitStrategy`: requires `HadoopConf` to create `HadoopFileSystemProxy`
**Problem Description**:
To reuse `AccordingToSplitSizeSplitStrategy`, LocalFile scenarios are also forced to introduce the `HadoopConf` concept. Although technically feasible (through `LocalFileHadoopConfig`), it is conceptually unnatural:
- LocalFile users typically don't care about Hadoop configuration
- Error messages may contain "Hadoop" wording, confusing LocalFile users
**Potential Risks**:
- **Risk 1**: User confusion ("Why do I need Hadoop configuration to read local files?")
- **Risk 2**: Misleading error messages (e.g., "Hadoop configuration error")
**Impact Scope**:
- **Direct Impact**: LocalFile Source user experience
- **Indirect Impact**: Documentation complexity
- **Affected Area**: LocalFile Connector
**Severity**: MINOR
**Improvement Suggestions**:
Option 1 (Recommended): Explicitly state in documentation
```markdown
For **LocalFile**, the `hadoop_conf` is optional and defaults to `file:///` schema.
You rarely need to configure it unless you have custom Hadoop settings.
```
Option 2 (Complex): Abstract `FileSystemProxy` interface
```java
public interface FileSystemProxy {
FSDataInputStream getInputStream(String path) throws IOException;
FileStatus getFileStatus(String path) throws IOException;
}
public class HadoopFileSystemProxy implements FileSystemProxy { ... }
public class LocalFileSystemProxy implements FileSystemProxy { ... } // Pure Java NIO
```
**Rationale**: Option 1 is simple and effective, Option 2 has clearer architecture but requires more work. The current solution is acceptable.
---
### Issue 8: Checkpoint Compatibility Not Fully Validated
**Location**:
`FileSourceSplit.java:44-49`
```java
public FileSourceSplit(String tableId, String filePath, long start, long length) {
this.tableId = tableId;
this.filePath = filePath;
this.start = start; // ← New field assigned in constructor
this.length = length; // ← New field assigned in constructor
}
```
**Related Context**:
- `serialVersionUID = 1L`
- Old version `FileSourceSplit` may only have `tableId`/`filePath`, `start`/`length` use default values 0/-1
**Problem Description**:
The PR did not validate the behavior of restoring from old version (without file split feature) checkpoints to the new version:
1. After deserializing `FileSourceSplit` from old checkpoint, `start=0, length=-1`
2. New code will read by byte range, `length=-1` may cause problems
3. Need to check if `AbstractReadStrategy.actualRead()` correctly handles `length=-1`
**Validation**:
Check `AbstractReadStrategy.safeSlice()`:
```java
protected static InputStream safeSlice(InputStream in, long start, long length) throws IOException {
// ...
return new BoundedInputStream(in, length); // How does BoundedInputStream handle length=-1?
}
```
Apache Commons IO's `BoundedInputStream(-1)` will read to the end of the stream, so the behavior is correct. But this is an implicit dependency.
**Potential Risks**:
- **Risk 1**: Jobs upgrading from 2.6.x to 2.7.x, after checkpoint recovery, data inconsistency
- **Risk 2**: The semantics of `length=-1` are not clear enough (does it mean "read to end of file" or "disable split"?)
**Impact Scope**:
- **Direct Impact**: Job recovery for upgrading users
- **Indirect Impact**: Data consistency
- **Affected Area**: All jobs using File Source
**Severity**: MAJOR
**Improvement Suggestions**:
```java
// FileSourceSplit.java
public FileSourceSplit(String tableId, String filePath, long start, long length) {
this.tableId = tableId;
this.filePath = filePath;
this.start = start;
// Clarify semantics: -1 means "entire file" (compatible with old version)
this.length = length;
}
// AbstractReadStrategy.java
protected static InputStream safeSlice(InputStream in, long start, long length) throws IOException {
if (start > 0) {
if (in instanceof Seekable) {
((Seekable) in).seek(start);
} else {
// ...
}
}
// length=-1 means unlimited (compatible with old checkpoint)
if (length < 0) {
return in;
}
return new BoundedInputStream(in, length);
}
```
Also add tests:
```java
@Test
void testDeserializeFromOldCheckpoint() {
// Simulate FileSourceSplit from old version serialization
FileSourceSplit oldSplit = new FileSourceSplit("table", "path"); // start=0, length=-1
// Verify reading entire file
List<String> records = readBySplit(oldSplit);
Assertions.assertEquals(fullFileRecords, records);
}
```
**Rationale**: Clarify the semantics of `length=-1`, add tests to verify upgrade compatibility.
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org