You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhiliang-wu (via GitHub)" <gi...@apache.org> on 2026/06/18 22:11:59 UTC

[PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

zhiliang-wu opened a new pull request, #11131:
URL: https://github.com/apache/seatunnel/pull/11131

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   Currently, the MaxcomputeSource divides the total record count into massive contiguous blocks sequentially (e.g., Reader 0 gets the first 25% of the table, Reader 1 gets the next 25%). Furthermore, it stores these splits internally using a HashSet, which completely scrambles the split assignment order before they are processed by the readers.
   
   Because of this, if a user uses ORDER BY or RANGE CLUSTERED BY in their MaxCompute table to prioritize data, running SeaTunnel with parallelism > 1 destroys this ordering entirely.
   
   This PR introduces a Round-Robin Split Assignment logic:
   
   Replaces the massive contiguous block chunking with striped/round-robin chunking (e.g. Reader 0 gets chunk 1, Reader 1 gets chunk 2). This allows all parallel readers to pull from the "top" of the table simultaneously, preserving a high-level priority order across multiple threads.
   Replaces HashSet with LinkedHashSet in MaxcomputeSourceSplitEnumerator to guarantee that the chronological insertion order of splits is strictly preserved when assigning them to the readers.
   
   ### Does this PR introduce _any_ user-facing change?
   
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
   If no, write 'No'.
   If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
   -->
   No. This PR only modifies the internal split enumeration and distribution logic for the MaxCompute Source. It introduces no new configuration options and does not break any existing contracts. It purely optimizes the reading order to be more evenly distributed across parallel readers.
   
   ### How was this patch tested?
   
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
   -->
   Ran mvn clean install -pl seatunnel-connectors-v2/connector-maxcompute -am -DskipTests to verify compilation and dependency integrity.
   Manually tested the execution graph locally to verify that splits are correctly striped and assigned in round-robin order without hash scrambling.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "zhangshenghang (via GitHub)" <gi...@apache.org>.
zhangshenghang commented on PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131#issuecomment-4763661284

   Nice cleanup. The round-robin striping plus `LinkedHashSet` reads much clearer than the old contiguous-block math, and switching the loop counter to `long` is a welcome side benefit — the old `int readerStart = i * splitRowNum` could silently overflow on very large tables, and the new splits are cleanly aligned to `split_row` instead of being clipped at reader boundaries. Two things I'd like to see addressed before merge.
   
   **1. The tests don't actually cover the thing this PR fixes.**
   
   The whole point of the change is preserving order, but `MaxcomputeSourceSplitTest` only asserts how many splits land on each reader — nothing checks that splits arrive in ascending `rowStart` per reader, which is the behavior the `LinkedHashSet` swap is there to guarantee. A future refactor could quietly reintroduce a `HashSet` and these tests would stay green.
   
   The tests also re-implement the `discoverySplits` loop inline rather than driving the enumerator, so they validate a copy of the logic, not the production path. At minimum, could you add an assertion that the splits collected per reader are in ascending `rowStart` order? That's the regression guard that matches the PR's intent.
   
   **2. Resuming an in-flight job from an older checkpoint will re-read data.**
   
   `discoverySplits()` dedups against restored state with `assignedSplits.forEach(splits::remove)`, which relies on split equality (all fields, including `rowStart` and `index`). This PR changes both the `rowStart` boundaries and the `index` assignment scheme, so a job checkpointed on a pre-change build and resumed on this one won't match any restored split — every split gets regenerated and reassigned, on top of the splits each reader restores from its own state, so previously-consumed rows get read again.
   
   Within a single build it stays deterministic and restore is fine, so this is strictly an upgrade-time concern. But since the PR description says "no user-facing change," it'd be worth at least calling out that resuming an interrupted job across this upgrade re-reads data. Not necessarily a blocker, just shouldn't be silent.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131#issuecomment-4763916290

   Thanks for calling this out, @zhangshenghang — I rechecked the current head and I agree your concern is a useful one.
   
   My earlier approval was based on the production logic itself: the current `chunkIndex` scope does fix the cross-table round-robin bug I raised before. So from the runtime-behavior side, I do not see a new correctness regression on the same commit.
   
   That said, your testing point is fair: the current test would be stronger if it asserted the per-reader `rowStart` ordering directly, and even better if the regression guard exercised the production enumerator path rather than a copied loop shape.
   
   Since there is still no new commit on this PR, I’m treating this as a reply-only follow-up rather than opening a new full review round on the same revision. If @zhiliang-wu pushes a small test-only follow-up for the ordering assertion / production-path coverage, I’m happy to re-review the updated head.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "zhiliang-wu (via GitHub)" <gi...@apache.org>.
zhiliang-wu commented on PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131#issuecomment-4749444760

   Thank you for your review. I've just implemented your exact suggested fix:
   
   1. Moved the int chunkIndex = 0; declaration to the top of the discoverySplits method, outside of the sourceTableInfos.values() loop so the round-robin distribution persists seamlessly across all tables.
   2. Wrote a new unit test testMultiTableRoundRobinDistribution that simulates exactly the scenario you described (multiple tables pushing 1 or 2 splits each) and validates that the splits are still evenly distributed among the readers across the entire workload footprint.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "zhiliang-wu (via GitHub)" <gi...@apache.org>.
zhiliang-wu commented on PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131#issuecomment-4767135454

    Thanks for the great feedback! You made excellent points regarding the tests only verifying a copy of the logic
    I've pushed a new commit to address these issues:
    
    1. **Tests Now Drive Production Logic**: To fix this without requiring mocked MaxCompute network sessions, I refactored MaxcomputeSourceSplitEnumerator. The core chunking logic has been extracted out of discoverySplits() into a package-private static method computeSplits(). All unit tests now invoke MaxcomputeSourceSplitEnumerator.computeSplits(...) directly, guaranteeing we are testing the actual production path.
   
   2. **Added Ascending Order Regression Guard**: I've updated MaxcomputeSourceSplitTest (specifically testComputeSplitsAscendingRowStartOrder) to strictly assert that the splits assigned to any given reader for any given table arrive in purely ascending rowStart order. If someone accidentally swaps the LinkedHashSet back out for a HashSet in the future, the test will immediately fail the build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "zhiliang-wu (via GitHub)" <gi...@apache.org>.
zhiliang-wu commented on PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131#issuecomment-4750974592

   It is the same commit but I have updated it with the moved chunkIndex scope and the multi-table regression test.
   Please check the PR activity for the updates


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131#issuecomment-4750795005

   Thanks for the update, and the intended fix sounds aligned with the blocker I called out.
   
   I checked the PR activity before replying here, and I still do not see a new commit on the current head yet. Since the code itself has not changed on GitHub, I am keeping this as a reply-only follow-up for now rather than starting a new full review round on the same version.
   
   Once you push the updated commit with the moved `chunkIndex` scope and the multi-table regression test, please ping me again and I will re-review the latest head from scratch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2] Maxcompute Source Round-Robin Split Assignment [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11131:
URL: https://github.com/apache/seatunnel/pull/11131


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org