You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/05/26 11:48:00 UTC
[GitHub] [incubator-seatunnel] Interest1-wyt opened a new pull request, #1958: [Feature] Add etl for data handler
Interest1-wyt opened a new pull request, #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-seatunnel/issues/1957).
- Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
-->
## Purpose of this pull request
<!-- Describe the purpose of this pull request. For example: This pull request adds checkstyle plugin.-->
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1176916582
> Generally looks good to me, It might be better if you can submit a PR to add e2e case for this plugin.
I try to add a test case to the e2e module, but the error "Could not find a valid Docker environment" is displayed when I run it. Running other test methods in e2e also reports this error. Why is this and how can I fix it?
The project runs on the win10 operating system and the docker component is installed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on a diff in pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on code in PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#discussion_r896331830
##########
docs/en/transform/nulltf.md:
##########
@@ -0,0 +1,67 @@
+# Nulltf
+
+## Description
+
+set default value for null field
+
+:::tip
+
+This transform only supported by engine Spark.
+
+:::
+
+## Options
+
+| name | type | required | default value |
+| ------------------- | ------- | -------- | ------------- |
+| fields | array | no | - |
+
+### fields [list]
+
+A list of fields whose default value will be set.
+The default value of the field can be set in the form of "=value" after the field name.
+If there is no "=value" after the field name, the default value will be set according to the field type.
+
+## Examples
+
+the configuration
+
+```bash
+ nulltf {
+ fields = ["name","price=0","num=100","flag","dt_timestamp=2022-05-18 13:51:40.603","dt_date=2022-05-19"]
+ }
Review Comment:
```suggestion
fields : {
price:0
}
}
```
Can you use the conf style here, then you don't need to parse "=".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on pull request #1958: [Feature] Add etl for data handler
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1138564600
Please add an issue and describe why you need to add this transform plugin. In fact, I am confusion with the plugin name `etl`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1139389364
>Dead Link Checker failed,It doesn't seem to be my problem. Can you help me rerun the ci
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1173045586
Please fix the scala checkstyle problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs merged pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
CalvinKirs merged PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1176040093
OK, I will revise it and submit it later :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1176201132
Sorry, I accidentally clicked the close button, could you please help to restart the build :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt closed pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt closed pull request #1958: [Feature][transform] Add a module to set default value for null field
URL: https://github.com/apache/incubator-seatunnel/pull/1958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] wuchunfu commented on a diff in pull request #1958: [Feature] Add etl for data handler
Posted by GitBox <gi...@apache.org>.
wuchunfu commented on code in PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#discussion_r882597103
##########
seatunnel-transforms/seatunnel-transforms-spark/seatunnel-transform-spark-etl/src/main/scala/org/apache/seatunnel/spark/transform/service/EtlHandler.scala:
##########
@@ -0,0 +1,8 @@
+package org.apache.seatunnel.spark.transform.service
+
Review Comment:
@Interest1-wyt Please add the license header, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on a diff in pull request #1958: [Feature] Add etl for data handler
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on code in PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#discussion_r883195194
##########
seatunnel-transforms/seatunnel-transforms-spark/seatunnel-transform-spark-etl/src/main/scala/org/apache/seatunnel/spark/transform/service/EtlHandler.scala:
##########
@@ -0,0 +1,8 @@
+package org.apache.seatunnel.spark.transform.service
+
Review Comment:
ok i'll add it later :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1176940536
> > Generally looks good to me, It might be better if you can submit a PR to add e2e case for this plugin.
> > I try to add a test case to the e2e module, but the error "Could not find a valid Docker environment" is displayed when I run it. Running other test methods in e2e also reports this error. Why is this and how can I fix it?
> > The project runs on the win10 operating system and the docker component is installed.
>
> 
Sorry, this is a problem with my local docker installation, which has been solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#issuecomment-1176190776
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] Interest1-wyt commented on a diff in pull request #1958: [Feature][transform] Add a module to set default value for null field
Posted by GitBox <gi...@apache.org>.
Interest1-wyt commented on code in PR #1958:
URL: https://github.com/apache/incubator-seatunnel/pull/1958#discussion_r907302004
##########
docs/en/transform/nulltf.md:
##########
@@ -0,0 +1,67 @@
+# Nulltf
+
+## Description
+
+set default value for null field
+
+:::tip
+
+This transform only supported by engine Spark.
+
+:::
+
+## Options
+
+| name | type | required | default value |
+| ------------------- | ------- | -------- | ------------- |
+| fields | array | no | - |
+
+### fields [list]
+
+A list of fields whose default value will be set.
+The default value of the field can be set in the form of "=value" after the field name.
+If there is no "=value" after the field name, the default value will be set according to the field type.
+
+## Examples
+
+the configuration
+
+```bash
+ nulltf {
+ fields = ["name","price=0","num=100","flag","dt_timestamp=2022-05-18 13:51:40.603","dt_date=2022-05-19"]
+ }
Review Comment:
Yes, I will resubmit later.
Thanks for your suggestion :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org