You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "chl-wxp (via GitHub)" <gi...@apache.org> on 2026/01/12 08:23:52 UTC
[PR] [Feature][connectors-v2/connector-file]Support logical ORC splits [seatunnel]
chl-wxp opened a new pull request, #10325:
URL: https://github.com/apache/seatunnel/pull/10325
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
### Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
If no, write 'No'.
If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
-->
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][connectors-v2/connector-file]Support logical ORC splits [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #10325:
URL: https://github.com/apache/seatunnel/pull/10325#issuecomment-4278479895
Thanks for working on this. I reviewed the PR locally from branch `seatunnel-review-10325` at commit `25aded2524e2d4a2805640e3b22a43d755f600fe`, against `upstream/dev` with merge base `81d873301d5d0bad16a3ef1680a3172d0968b8d9`.
The feature is useful: ORC should be split on logical stripe boundaries rather than raw bytes, and this PR moves in that direction. There are two merge-blocking problems, though: the branch now conflicts with the current file split architecture on `dev`, and the ORC split read path drops the projected ORC schema.
## What this PR solves
- User pain point: large LocalFile ORC files are currently assigned as whole files, so a single large ORC file cannot benefit from source parallelism.
- Fix approach: add `OrcFileSplitStrategy`, generate `FileSourceSplit(start, length)` from ORC stripe metadata, select that strategy for LocalFile ORC, and make `OrcReadStrategy` read a split range.
- In one sentence: this PR adds logical ORC splitting for LocalFile source, but it needs a rebase plus a reader correctness fix before merge.
Simple example: if an ORC file has 10 stripes, this PR can produce several split ranges so multiple readers can process it. But if the user configures `read_columns = ["c_string"]`, the current split reader creates a new ORC `Reader.Options().range(...)` without the projected schema, so it can read with a different schema from the `VectorizedRowBatch` used for conversion.
## Complete runtime path
```text
LocalFile ORC source startup
-> LocalFileSource(ReadonlyConfig)
-> LocalFileSplitStrategyFactory.initFileSplitStrategy(readonlyConfig) [LocalFileSplitStrategyFactory.java:33-63]
-> enable_file_split=true
-> file_format_type=ORC
-> new OrcFileSplitStrategy(file_split_size)
Split enumeration
-> BaseMultipleTableFileSource.createEnumerator(...) [BaseMultipleTableFileSource.java:83-87]
-> MultipleTableFileSourceSplitEnumerator.open() [MultipleTableFileSourceSplitEnumerator.java:81-89]
-> fileSplitStrategy.split(tableId, filePath)
-> OrcFileSplitStrategy.split(...) [OrcFileSplitStrategy.java:61-67]
-> readStripes(filePath)
-> OrcFile.createReader(path, readerOptions)
-> reader.getStripes()
-> splitByStripes(...)
-> generate FileSourceSplit(tableId, filePath, start, length)
-> assignSplit(taskId)
-> parallelism=1: all splits go to one reader
-> parallelism>1: splits are assigned round-robin
Reader phase
-> MultipleTableFileSourceReader.pollNext(...) [MultipleTableFileSourceReader.java:68-85]
-> readStrategyMap.get(split.getTableId())
-> readStrategy.read(split, output)
-> OrcReadStrategy.read(FileSourceSplit, output) [OrcReadStrategy.java:92-160]
-> build SeaTunnel projected ORC schema
-> rows = reader.rows(reader.options().schema(schema))
-> if useSplitRange=true
-> options = new Reader.Options().range(start, length)
-> rows = reader.rows(options) // projected schema is dropped here
-> rowBatch = schema.createRowBatch()
-> convert rowBatch columns with SeaTunnel row type
```
## Findings
Issue 1: the PR has real conflicts with current `dev` in the core file split path
- Location: `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/AccordingToSplitSizeSplitStrategy.java:46`; `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSourceSplit.java:52`; `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/ParquetFileSplitStrategy.java:81`; `seatunnel-connectors-v2/connector-file/connector-file-local/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/local/source/split/LocalFileSplitStrategyFactory.java:31`
- Severity: High
- Description: GitHub reports this branch as conflicting, and I reproduced it locally in a temporary worktree. The merge conflicts are not just in tests or docs: they are in `AccordingToSplitSizeSplitStrategy`, `FileSourceSplit`, `ParquetFileSplitStrategy`, `LocalFileSourceTest`, and a modify/delete conflict for `LocalFileSplitStrategyFactory`, which no longer exists on current `dev`.
- Risk: this cannot be merged as-is. A careless conflict resolution could also discard current `dev` work around split strategy aggregation and `FileSourceSplit` checkpoint compatibility.
- Suggested fix: rebase onto the latest `dev` and apply the ORC split support on top of the current file split architecture. Please keep the current `FileSourceSplit` compatibility logic and avoid restoring the deleted LocalFile-only split strategy factory.
Issue 2: ORC split reading drops the projected schema
- Location: `seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/reader/OrcReadStrategy.java:120`
- Severity: High
- Description: the non-split path opens the ORC reader with `reader.options().schema(schema)`, where `schema` is built from the SeaTunnel row type. The split path then replaces the record reader with `reader.rows(new Reader.Options().range(start, length))`, losing that schema. The following code still uses `schema.createRowBatch()`, `children.get(j)`, and `seaTunnelRowType.getFieldType(j)` for conversion.
- Risk: default full-column reads may appear to work, but `read_columns`, explicit schema, casts, or any schema mismatch can produce wrong columns, conversion errors, or lost data in split mode.
- Suggested fix: use one `Reader.Options` instance that always carries the projected schema, and add the range only when needed, for example:
```java
Reader.Options options = reader.options().schema(schema);
if (useSplitRange) {
options.range(split.getStart(), split.getLength());
}
try (RecordReader rows = reader.rows(options)) {
...
}
```
Please also add ORC split coverage with projection/read_columns and compare split vs non-split output.
Issue 3: the new ORC split E2E does not prove parallel split correctness
- Location: `seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/orc/local_file_orc_enable_split_to_assert.conf:18`; `seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e/src/test/resources/orc/local_file_orc_enable_split_to_assert.conf:45`
- Severity: Medium
- Description: the E2E sets `parallelism = 1`, so it does not validate multi-reader split assignment. It also uses only `MAX_ROW=5`; if the split path drops rows but still emits no more than 5 rows, the row-count assertion can still pass.
- Risk: the test may pass while ORC split loses rows, duplicates rows, or fails only under parallel assignment.
- Suggested fix: use parallelism greater than 1 and assert the exact expected row count, either with both min/max row rules or an equivalent exact-count rule. A projection split E2E would also catch Issue 2.
## Compatibility
For users who do not enable `enable_file_split`, ORC behavior should remain compatible. For ORC split users, this is new behavior. The important compatibility point is checkpoint/restore: current `dev` already has `FileSourceSplit` compatibility handling, so the rebased PR should preserve it and only add ORC-specific strategy/reader behavior.
## Performance and side effects
Reading ORC stripe metadata is a reasonable startup cost for splitting large files. The split list size grows with stripe count, which is expected. Since this PR currently targets LocalFile only, I do not see a remote filesystem/authentication issue in the submitted path. If this is later extended to HDFS/S3/OSS, the stripe reader should use the corresponding Hadoop configuration/proxy instead of a plain `new Configuration()`.
## Tests and docs
The new `OrcFileSplitStrategyTest` is useful for pure split calculation. The missing coverage is reader correctness: split + projection, split vs non-split equivalence, parallel assignment, and restore behavior. The docs/en and docs/zh LocalFile source updates are aligned, but the implementation needs to be fixed before the docs promise is safe.
## Merge conclusion
### Conclusion: can be merged after fixes
1. Blocking items that must be fixed
- Issue 1: rebase and resolve the current file split conflicts on top of latest `dev`.
- Issue 2: preserve the projected ORC schema when opening the range reader for split mode.
2. Suggested non-blocking improvements
- Issue 3: strengthen the ORC split E2E with parallelism > 1 and exact row-count/projection assertions.
Overall, the feature is worth continuing and the stripe-based approach is the right shape. After the rebase and the schema/range reader fix, this should be much closer to merge-ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][connectors-v2/connector-file]Support logical ORC splits [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #10325: [Feature][connectors-v2/connector-file]Support logical ORC splits
URL: https://github.com/apache/seatunnel/pull/10325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][connectors-v2/connector-file]Support logical ORC splits [seatunnel]
Posted by "chl-wxp (via GitHub)" <gi...@apache.org>.
chl-wxp commented on PR #10325:
URL: https://github.com/apache/seatunnel/pull/10325#issuecomment-3742003739
https://github.com/apache/seatunnel/issues/10129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][connectors-v2/connector-file]Support logical ORC splits [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10325:
URL: https://github.com/apache/seatunnel/pull/10325#issuecomment-5502682976
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][connectors-v2/connector-file]Support logical ORC splits [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10325:
URL: https://github.com/apache/seatunnel/pull/10325#issuecomment-5335916625
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org