You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Adamyuanyuan (via GitHub)" <gi...@apache.org> on 2025/08/05 06:29:34 UTC
[PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Adamyuanyuan opened a new pull request, #9664:
URL: https://github.com/apache/seatunnel/pull/9664
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
### Purpose of this pull request
X2SeaTunnel is a tool for converting DataX and other configuration files to SeaTunnel configuration files, designed to help users quickly migrate from other data integration platforms to SeaTunnel.
This is the implementation of the first version.
### Does this PR introduce _any_ user-facing change?
Added a new tool with the following functions:
- ✅ **Standard Configuration Conversion**: DataX → SeaTunnel configuration file conversion
- ✅ **Custom Template Conversion**: Support for user-defined conversion templates
- ✅ **Detailed Conversion Reports**: Generate Markdown format conversion reports
- ✅ **Regular Expression Variable Extraction**: Extract variables from configuration using regex, supporting custom scenarios
- ✅ **Batch Conversion Mode**: Support directory and file wildcard batch conversion, automatic report and summary report generation
### Basic Usage
```bash
# Standard conversion: Use default template system with built-in common Sources and Sinks
./bin/x2seatunnel.sh -s examples/source/datax-mysql2hdfs.json -t examples/target/mysql2hdfs-result.conf -r examples/report/mysql2hdfs-report.md
# Custom task: Implement customized conversion requirements through custom templates
# Scenario: MySQL → Hive (DataX doesn't have HiveWriter)
# DataX configuration: MySQL → HDFS Custom task: Convert to MySQL → Hive
./bin/x2seatunnel.sh -s examples/source/datax-mysql2hdfs2hive.json -t examples/target/mysql2hive-result.conf -r examples/report/mysql2hive-report.md -T templates/datax/custom/mysql-to-hive.conf
# YAML configuration method (equivalent to above command line parameters)
./bin/x2seatunnel.sh -c examples/yaml/datax-mysql2hdfs2hive.yaml
# Batch conversion mode: Process by directory
./bin/x2seatunnel.sh -d examples/source -o examples/target2 -R examples/report2
# Batch mode supports wildcard filtering
./bin/x2seatunnel.sh -d examples/source -o examples/target3 -R examples/report3 --pattern "*-full.json" --verbose
# View help
./bin/x2seatunnel.sh --help
```
### Conversion Report
After conversion is completed, view the generated Markdown report file, which includes:
- **Basic Information**: Conversion time, source/target file paths, connector types, conversion status, etc.
- **Conversion Statistics**: Counts and percentages of direct mappings, smart transformations, default values used, and unmapped fields
- **Detailed Field Mapping Relationships**: Source values, target values, filters used for each field
- **Default Value Usage**: List of all fields using default values
- **Unmapped Fields**: Fields present in DataX but not converted
- **Possible Error and Warning Information**: Issue prompts during conversion process
For batch conversions, a batch summary report `summary.md` will be generated in the batch report directory, including:
- **Conversion Overview**: Overall statistics, success rate, duration, etc.
- **Successful Conversion List**: Complete list of successfully converted files
- **Failed Conversion List**: Failed files and error messages (if any)
### How was this patch tested?
Yes, including unit tests, local tests, and production tests.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3243605985
Move to https://github.com/apache/seatunnel-tools/pull/2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3157254455
> > I think we should create new sub repository just like mcp. cc @davidzollo @TyrantLucifer
>
> We need to reach a consensus in the email, please send a discussion in dev @Adamyuanyuan
Yes, we need to discuss to create a new repository for this PR.
@Adamyuanyuan you did a amazing job. Please send a discussion in dev@seatunnel.apache.org, then we can create a new repository, thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3157215621
> I think we should create new sub repository just like mcp. cc @davidzollo @TyrantLucifer
We need to reach a consensus in the email, please send a discussion in dev @Adamyuanyuan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Adamyuanyuan (via GitHub)" <gi...@apache.org>.
Adamyuanyuan commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3158465056
> I think we should create new sub repository just like mcp. cc @davidzollo @TyrantLucifer
We need to first discuss the necessity of adding a new repo. Is it better to integrate it into the Seatunnel project or have it as a separate project?
1. If we add a new repo, it will be inconvenient for me to reuse the current code management solution. Also, there might be a lot of work in building a new project, and I don't have much experience in this regard.
2. Previously, we chose Java for implementation considering the reuse of the technical stack of the current Seatunnel project. If it's a new project, should we consider Python? Each has its advantages and disadvantages, but Python generally has fewer lines of code and better support for engines like Jinja2.
3. If it's a separate project, the functionality of X2Seatunnel seems a bit thin, and I'm worried that the project value might not be sufficient.
Of course, the advantages of a separate project are obvious. It can be iterated faster and is easier to control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3154114040
I think we should create new sub repository just like mcp. cc @davidzollo @TyrantLucifer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Adamyuanyuan (via GitHub)" <gi...@apache.org>.
Adamyuanyuan commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3153692420
#9507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X closed pull request #9664: [Feature][tools] X2SeaTunnel #9507
URL: https://github.com/apache/seatunnel/pull/9664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][tools] X2SeaTunnel #9507 [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9664:
URL: https://github.com/apache/seatunnel/pull/9664#issuecomment-3187364815
Hi @Adamyuanyuan ,
We created an new repo https://github.com/apache/seatunnel-tools.
Please raise a PR to commit the code of x2seatunnel.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org