You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2021/12/30 07:59:45 UTC
[GitHub] [incubator-seatunnel] leo65535 opened a new issue #904: [Feature][config] improve the config parse logic
leo65535 opened a new issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904
### Search before asking
- [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
improve the config parse logic
### Usage Scenario
improve the config parse logic
### Related issues
improve the config parse logic
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1003287798
Hi @RickyHuo @garyelephant @CalvinKirs, I implemented the second version, please review the patch when you are free, thanks.
1. Add `SeaTunnelConfigFactory` which works as `ConfigFactory`.
2. Extract the sort logic to `SeaTunnelConfigParser`(Keep it the same as before)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 edited a comment on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 edited a comment on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1002929651
cc @garyelephant @CalvinKirs @simon824, I had rewrite the logic and raise a draft pull request, it will help to understand my statement.
had tested the following application.conf
```
./bin/start-seatunnel-spark.sh -c config/application.conf -e client -m local -i app=test
```
```
env {
"spark.app.name" = "SeaTunnel"
"spark.executor.instances" = 2
"spark.executor.cores" = 1
"spark.executor.memory" = "1g"
"spark.streaming.batchDuration" = 5
}
source {
fakeStream {
content = ["Hello World, InterestingLab"]
}
}
transform {
}
sink {
Console {}
}
====================================================
env {
"spark.app.name" = "SeaTunnel"
"spark.executor.instances" = 2
"spark.executor.cores" = 1
"spark.executor.memory" = "1g"
"spark.streaming.batchDuration" = 5
}
source {
"org.apache.seatunnel.spark.source.FakeStream" {
content = ["Hello World, InterestingLab"]
}
}
transform {
}
sink {
Console {}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1003655036
> can you briefly explain how to implement the config parsing logic to keep the order as in config file?
I move your changes to `SeaTunnelConfigParser`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 edited a comment on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 edited a comment on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1002929651
cc @garyelephant @CalvinKirs @simon824, I had rewrite the logic and raise a draft pull request, it will help to understand my statement.
Had tested the following application.conf
```
./bin/start-seatunnel-spark.sh -c config/application.conf -e client -m local -i app=test
```
```
env {
"spark.app.name" = "SeaTunnel"
"spark.executor.instances" = 2
"spark.executor.cores" = 1
"spark.executor.memory" = "1g"
"spark.streaming.batchDuration" = 5
}
source {
fakeStream {
content = ["Hello World, InterestingLab"]
}
}
transform {
}
sink {
Console {}
}
====================================================
env {
"spark.app.name" = "SeaTunnel"
"spark.executor.instances" = 2
"spark.executor.cores" = 1
"spark.executor.memory" = "1g"
"spark.streaming.batchDuration" = 5
}
source {
"org.apache.seatunnel.spark.source.FakeStream" {
content = ["Hello World, InterestingLab"]
}
}
transform {
}
sink {
Console {}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] RickyHuo commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
RickyHuo commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1002949926
I have saw the draft PR. The seatunnel-config has been replaced with typesafe config.
The reason why we develop seatunnel-config base on typesafe config is that the plugin order of parsed Config is not the same as our configuration when transform section has multiply plugins.
The origin issue is #29, I'm sorry I forget config test at that moment.
We should test those demo:
```
source {
}
transform {
sql {
sql = "sql1"
}
sql {
sql = "sql2"
}
sql {
sql = "sql3"
}
}
sink {
}
```
```
source {
}
transform {
sql {
sql = "sql1"
}
json {
source_field = "xxx"
}
sql {
sql = "sql3"
}
}
sink {
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] garyelephant commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
garyelephant commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1003326985
> Hi @RickyHuo @garyelephant @CalvinKirs, I implemented the second version, please review the patch when you are free, thanks.
>
> 1. Add `SeaTunnelConfigFactory` which works as `ConfigFactory`.
> 2. Extract the sort logic to `SeaTunnelConfigParser`(Keep it the same as before)
>
> 
@leo65535 thanks for your contribution, can you briefly explain how to implement the config parsing logic to keep the order as in config file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1002929651
cc @garyelephant @CalvinKirs @simon824, I had rewrite the logic and raise a draft pull request, it will help to understand my statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] simon824 commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1002943061
Great job ! I'm ok with these two changes, and I will test more cases later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 commented on issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 commented on issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904#issuecomment-1002990663
> our configuration when transform section has multiply plugins.
Thanks for the warning, and that's a real problem. Let me think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] leo65535 closed issue #904: [Feature][config] improve the config parse logic
Posted by GitBox <gi...@apache.org>.
leo65535 closed issue #904:
URL: https://github.com/apache/incubator-seatunnel/issues/904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org