You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "MyeoungDev (via GitHub)" <gi...@apache.org> on 2026/06/29 16:00:24 UTC
[PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
MyeoungDev opened a new pull request, #11222:
URL: https://github.com/apache/seatunnel/pull/11222
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
### Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
Close #11221
### Description
This PR expands CopyTransform support for data types that are already used in SeaTunnel pipelines, including
- `timestamp_tz`
- vector types
- `binary_vector`
- `float_vector`
- `float16_vector`
- `bfloat16_vector`
- `sparse_float_vector`
- `multiple_row`
### What changed
- Added support for `timestamp_tz` in CopyTransform
- Added support for `vector types` in CopyTransform
- Added support for `multiple_row` in CopyTransform
- Added unit tests for supported copy cases
- Added E2E coverage for the Copy path
- Updated CopyTransform documentation in both English and Chinese
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released SeaTunnel versions or within the unreleased branches such as dev.
If no, write 'No'.
If you are adding/modifying connector documents, please follow our new specifications: https://github.com/apache/seatunnel/issues/4544.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If you are adding E2E test cases, maybe refer to https://github.com/apache/seatunnel/blob/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/mysqlcdc_to_mysql.conf, here is a good example.
-->
- `./mvnw -pl seatunnel-transforms-v2 -Dskip.spotless=true -Dtest=CopyFieldTransformTest test`
- `TestCopyIT` e2e passed locally
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "MyeoungDev (via GitHub)" <gi...@apache.org>.
MyeoungDev commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-4857227975
Hi @davidzollo, @nzw921rx, @DanielLeens,
while working on this PR, I found that the Spark E2E path may need additional Spark Translation layer support for vector types.
Spark E2E executes this scenario through the Spark translation layer, and I confirmed that the current Spark translation layer does not support vector types yet.
The relevant classes I found are
- `org.apache.seatunnel.translation.spark.utils.TypeConverterUtils`
- `org.apache.seatunnel.translation.spark.serialization.InternalRowConverter`
- `org.apache.seatunnel.translation.spark.serialization.SeaTunnelRowConverter`
Could you please confirm whether this understanding is correct?
If so, should I add Spark Translation Layer support for vector types in this PR, or keep this PR focused on CopyTransform and open a separate PR for Spark vector type support?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-4854223554
Thanks for the update. I rechecked the current head against the latest `dev`, and the source-level blockers from my previous Daniel review still stand on this branch.
- base branch: `dev`
- head SHA: `0dd4fef49720a4340c0a576103cca6cadc302a82`
- compare status: `diverged`
- ahead_by: `2`
- behind_by: `12`
- CI / merge gate snapshot: `Build FAILURE`
The current failing CI signal does not look actionable on top of this stale head yet, so the lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I am happy to take another look on top of the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-4899183978
Thanks for the update. I checked the current head against the latest `dev` again.
- base branch: `dev`
- head SHA: `a2630197ce59d0834dfe10d9aab1711732944543`
- compare status: `diverged`
- ahead_by: `4`
- behind_by: `80`
- current failing signal: `Build`
On this round, the visible failing CI signal still does not look directly tied to the current PR-owned diff from Daniel's side, and the branch is still behind upstream. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If the same lane still fails on the updated head afterwards, please drop the new failing check link here and I can help narrow it down against the rebased revision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "SEZ9 (via GitHub) via commits" <co...@seatunnel.apache.org>.
SEZ9 commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6818356387
@MyeoungDev thanks for the update and for walking through the changes.
- **Vector copy (full capacity, position 0):** Agreed on the approach. Since `VectorUtils.toFloatArray` reads by capacity, preserving the source position would just shift the underflow problem; a fresh heap buffer with the whole capacity and position 0 is what downstream consumers expect. Glad to hear the moved-position, direct-buffer and byte-order cases now have unit tests and that the vector E2E checks the copy via `VECTOR_DIMS` and `L2_DISTANCE`.
- **`multiple_row` removal:** No need to bring it back. If it can't be declared as a column type and multi-table Copy already works without it, dropping the branch is cleaner than keeping dead code.
- **Docs, `instanceof` guards, Assert comment, indentation:** Noted, thanks.
I'll go through the latest changes and verify each of the earlier points against the diff, in particular that the Copy doc body describes the deep-copy semantics for bytes/vectors and that the `instanceof` fallback raises the structured `CommonError` with field context rather than a bare `ClassCastException`. Will follow up once I've done that pass.
<!-- streview-comment:2296 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-5119645274
Thanks again for the careful work on expanding `CopyTransform` support. This PR is still worth carrying forward because the copy path should handle `timestamp_tz`, vector values, and `multiple_row` values without shallow-copy surprises.
I helped refresh the original PR branch directly:
- merged the latest `apache/dev` into `MyeoungDev:improve/copytransform-extend-types`
- preserved the existing CopyTransform, assert-connector, E2E config, unit-test, and EN/ZH docs scope
- ran scoped Spotless for `seatunnel-transforms-v2` and `connector-assert`
- pushed the updated original branch to `dd2cbc0621b26436f8c204567a852c490d299a4b`
The PR is now mergeable from the sync/conflict side. The remaining gates are the normal GitHub CI / review gates, and the fresh workflow jobs are queued on the updated head.
Also, your earlier note about Spark vector translation still makes sense as a separate follow-up area. Keeping this PR focused on the CopyTransform path is the right shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6473942002
Sorry for not reconciling the vector consumer contracts in my earlier clearance. Thanks @SEZ9 for the concrete counterevidence. I rechecked CopyFieldTransform.cloneByteBuffer at lines 196-205, VectorUtils.toFloatArray/toShortArray at lines 57-83, and MilvusSinkConverter at lines 86-92. Copy allocates only remaining bytes, while those consumers use capacity or array(). A nonzero-position input therefore changes the copied dimension, and a direct binary-vector buffer cannot satisfy the array-backed Milvus path. Whether such an input is a supported source representation must be demonstrated, rather than assumed: some producer/converter paths already require a canonical full heap buffer. I am withdrawing the unconditional merge-ready wording while that contract is unresolved. +1 to resolving your buffer-contract concern with a test through the actual supported source -> Copy -> vector-consumer path, covering position/limit, direct/heap and byte order. Please either enforce and test th
e canonical input contract at its ownership boundary or provide a compatible copy/consumer contract. Green CI alone does not close this discussion; this is a correction to the prior conclusion, not a second partial full review or a new claim of runtime validation.
Gate snapshot (2026-10-28T15:59:52+00:00), head `ac877ada9dc2081f5f7df17cd677ece4209611c7`: [Build](https://github.com/apache/seatunnel/runs/122617822624) is `SUCCESS`.
The green check does not clear the independent source/documentation or scope requests in my [previous conclusion](https://github.com/apache/seatunnel/pull/11222#pullrequestreview-5676327685). Those requests remain unchanged; this is not an approval.
Comparison recorded in this scan: base `dev`, `diverged`, ahead 7 / behind 15, no merge conflict. Branch drift is metadata here and is not an additional sync request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-5149758683
Thanks for your contribution for Apache SeaTunnel.
To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
Slack workspace link: https://s.apache.org/seatunnel-slack
This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6412434890
Updated in `0123bcbf61eef992eecbc55a0171102ee3eeaa8f`.
Documented the three clone helpers: independent remaining-byte buffer storage with unchanged source position/limit, row metadata preservation, and rejection of missing or unknown nested table IDs.
This follow-up changes only comments/documentation. Scoped Spotless, static diff checks and independent source review passed; no local functional execution was performed. The remote head and complete file list were read back. Fresh current-head GitHub CI and repository maintainer gates remain separate; no runtime or merge approval is claimed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6393723910
Updated the original branch to `c48fd052e07ad480b85006dff91f5484f1797d3d` to address the reproducible Spark vector failure from the latest review.
`TestCopyIT.testCopy` continues to run on every engine with the supported scalar, nested-row and timestamp-with-time-zone scenarios. All five existing vector fields and their assertions now run in a separate `testCopyVectors` case on Flink and Zeta, since the Spark translation layer rejects `BINARY_VECTOR` before the Copy transform runs. This preserves the existing supported-engine coverage rather than disabling the entire Copy scenario on Spark.
Scoped Spotless and diff checks passed, and the patch received an independent source review. The remote file list and mergeability were verified: nine expected files, mergeable against `dev`. Fresh CI has not reported results on this head yet; runtime validation remains pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-5149756878
Thanks for your contribution for Apache SeaTunnel.
To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
Slack workspace link: https://s.apache.org/seatunnel-slack
This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join us😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6418111638
Good job.
Thanks a lot for your valuable contribution to Apache SeaTunnel!
To pass the automated CI checks for this PR, could you please enable CI workflows in your forked repository first? You can refer to the check run link below as a reference for the required CI configuration https://github.com/apache/seatunnel/actions/runs/40315069579
If you run into any trouble enabling CI or encounter related errors during the process, feel free to leave a comment here. Contributors are always ready to offer help~
<img width="2412" height="1472" alt="image" src="https://github.com/user-attachments/assets/6abd2ea1-da45-4d78-a81c-0158f2d7a851" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-4845333633
Thanks for the update. I rechecked the current head against the latest `dev`, and this branch is still behind upstream.
- base branch: `dev`
- head SHA: `0dd4fef49720a4340c0a576103cca6cadc302a82`
- compare status: `diverged`
- ahead_by: `2`
- behind_by: `4`
- mergeable_state: `unknown`
- CI / merge gate snapshot: `failing` (failing: `Build [failure]`)
Because the PR is still behind the latest base, any CI result and merge gate signal here is mixed with upstream drift. The lowest-cost next step is to sync with the latest `dev` and rerun CI first. If anything still fails after the sync, I'm happy to take another look on top of the updated head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "MyeoungDev (via GitHub) via commits" <co...@seatunnel.apache.org>.
MyeoungDev commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6785204904
Thanks @SEZ9 @DanielLeens. pushed an update.
Vectors are now copied as a full heap buffer (whole capacity, position 0). I didn't keep the source position as suggested, since `VectorUtils.toFloatArray` reads by capacity and would underflow. Added unit tests for moved position, direct buffer and byte order, and the vector E2E now checks the copy with `VECTOR_DIMS` and `L2_DISTANCE`.
Removed multiple_row. It can't be declared as a column type, and multi-table Copy already works without it. Happy to bring it back if you'd prefer.
Also added supported types and the Spark note to the docs, the instanceof guards, the Assert comment, and the indentation fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Transform-V2] Expand CopyTransform support for timestamp_tz, vector, and multiple_row types [seatunnel]
Posted by "DanielLeens (via GitHub) via commits" <co...@seatunnel.apache.org>.
DanielLeens commented on PR #11222:
URL: https://github.com/apache/seatunnel/pull/11222#issuecomment-6850975079
Thanks @SEZ9 for keeping the pending source verification explicit. My prior source conclusion remains clear, and I will not mark your independent verification complete on your behalf. The two concrete remaining actions from my review are unchanged: remove `multiple_row` from the title/description so they match the delivered scope, and take the already-merged SFTP image repair #13203 from latest `dev` before rerunning Build. This sync request is tied to the documented `atmoz/sftp:alpine-3.7` 404 failure, not merely to branch drift. No extra vector feature or unrelated dependency change is needed.
This is a discussion reply on unchanged head `a9b2a1dcdab92522affe2a7646ec4757567f04fb`, not another complete source review. Daniel has READ repository permission; effective maintainer approval/dismissal remains separate. No local tests were run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org