You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "QuakeWang (via GitHub)" <gi...@apache.org> on 2026/06/21 12:31:34 UTC

[PR] [Fix][Connector-V2] Flush Doris load before schema change [seatunnel]

QuakeWang opened a new pull request, #11156:
URL: https://github.com/apache/seatunnel/pull/11156

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   
   <!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
   
   Doris sink applied schema changes while the current Stream Load could still contain rows serialized with the old schema. Those rows could then be committed after the target table had already been altered, mixing old-schema and new-schema data in the same load. This is unsafe for positional formats such as CSV and for schema changes such as drop or rename columns.
   
   This PR flushes the in-flight Stream Load before applying schema changes in non-2PC mode, then opens a fresh load for rows written with the new schema.
   
   The 2PC path is intentionally unchanged because Doris 2PC currently uses one transaction and label per checkpoint. Flushing in the middle of a checkpoint would risk an orphan pre-commit transaction or checkpoint label reuse.
   
   ### Does this PR introduce _any_ user-facing change?
   
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
   If no, write 'No'.
   If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
   -->
   
   Yes.
   
   For Doris sink with `sink.enable-2pc = false`, buffered rows are now flushed before schema changes are applied. This prevents old-schema rows from being committed against the altered table.
   
   The schema evolution documentation now also clarifies the limitation of Doris schema evolution with `sink.enable-2pc = true`.
   
   ### How was this patch tested?
   
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
   -->
   
   Added unit tests for:
   
   - non-2PC schema-change ordering: `stopLoad -> apply DDL -> startLoad`
   - 2PC behavior: schema changes do not flush in the middle of a checkpoint
   
   Verified with:
   
   ```bash
   ./mvnw spotless:apply
   ./mvnw -pl seatunnel-connectors-v2/connector-doris -Dtest=DorisSinkWriterTest test
   ./mvnw -q -DskipTests verify
   ```
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [x] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Connector-V2] Flush Doris load before schema change [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11156:
URL: https://github.com/apache/seatunnel/pull/11156


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org