You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/01/24 11:47:20 UTC
[GitHub] [incubator-seatunnel] simon824 opened a new pull request #1154: [SeaTunnel #990][FOLLOWUP] Optimize doris connector checkConfig
simon824 opened a new pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154
## Purpose of this pull request
subTask of #1153
Optimize doris connector checkConfig
## Check list
* [ ] Code changed are covered with tests, or it does not need tests for reason:
* [ ] If any new Jar binary package adding in you PR, please add License Notice according
[New License Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/developement/NewLicenseGuide.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xtr1993 commented on pull request #1154: [Feature] [connector] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
xtr1993 commented on pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154#issuecomment-1035915649
> @xtr1993 PTAL,Thx
LGMT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] simon824 commented on a change in pull request #1154: [SeaTunnel #990][FOLLOWUP] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
simon824 commented on a change in pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154#discussion_r790667357
##########
File path: seatunnel-connectors/seatunnel-connector-spark-doris/src/main/scala/org/apache/seatunnel/spark/sink/Doris.scala
##########
@@ -68,35 +69,23 @@ class Doris extends SparkBatchSink with Serializable {
}
override def checkConfig(): CheckResult = {
- val checkResult =
- checkAllExists(config, Config.HOST, Config.DATABASE, Config.TABLE_NAME, Config.USER, Config.PASSWORD)
+ checkAllExists(config, HOST, DATABASE, TABLE_NAME, USER, PASSWORD)
+ }
- if (!checkResult.isSuccess) {
- checkResult
- } else if (config.hasPath(Config.USER) && !config.hasPath(Config.PASSWORD) || config.hasPath(
- Config.PASSWORD) && !config.hasPath(Config.USER)) {
- CheckResult.error(Config.CHECK_USER_ERROR)
- } else {
- val host: String = config.getString(Config.HOST)
- val dataBase: String = config.getString(Config.DATABASE)
- val tableName: String = config.getString(Config.TABLE_NAME)
- this.apiUrl = s"http://$host/api/$dataBase/$tableName/_stream_load"
- if (TypesafeConfigUtils.hasSubConfig(config, Config.ARGS_PREFIX)) {
- val properties = TypesafeConfigUtils.extractSubConfig(config, Config.ARGS_PREFIX, true)
- val iterator = properties.entrySet().iterator()
- while (iterator.hasNext) {
- val map = iterator.next()
- val split = map.getKey.split("\\.")
- if (split.size == 2) {
- propertiesMap.put(split(1), String.valueOf(map.getValue.unwrapped))
- }
Review comment:
it's better to move these to `prepare()` after `checkConfig()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs commented on pull request #1154: [SeaTunnel #990][FOLLOWUP] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
CalvinKirs commented on pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154#issuecomment-1022787644
@xtr1993 PTAL,Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] simon824 commented on a change in pull request #1154: [SeaTunnel #990][FOLLOWUP] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
simon824 commented on a change in pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154#discussion_r790667357
##########
File path: seatunnel-connectors/seatunnel-connector-spark-doris/src/main/scala/org/apache/seatunnel/spark/sink/Doris.scala
##########
@@ -68,35 +69,23 @@ class Doris extends SparkBatchSink with Serializable {
}
override def checkConfig(): CheckResult = {
- val checkResult =
- checkAllExists(config, Config.HOST, Config.DATABASE, Config.TABLE_NAME, Config.USER, Config.PASSWORD)
+ checkAllExists(config, HOST, DATABASE, TABLE_NAME, USER, PASSWORD)
+ }
- if (!checkResult.isSuccess) {
- checkResult
- } else if (config.hasPath(Config.USER) && !config.hasPath(Config.PASSWORD) || config.hasPath(
- Config.PASSWORD) && !config.hasPath(Config.USER)) {
- CheckResult.error(Config.CHECK_USER_ERROR)
- } else {
- val host: String = config.getString(Config.HOST)
- val dataBase: String = config.getString(Config.DATABASE)
- val tableName: String = config.getString(Config.TABLE_NAME)
- this.apiUrl = s"http://$host/api/$dataBase/$tableName/_stream_load"
- if (TypesafeConfigUtils.hasSubConfig(config, Config.ARGS_PREFIX)) {
- val properties = TypesafeConfigUtils.extractSubConfig(config, Config.ARGS_PREFIX, true)
- val iterator = properties.entrySet().iterator()
- while (iterator.hasNext) {
- val map = iterator.next()
- val split = map.getKey.split("\\.")
- if (split.size == 2) {
- propertiesMap.put(split(1), String.valueOf(map.getValue.unwrapped))
- }
Review comment:
move these from `checkConfig()` to `prepare()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] simon824 commented on a change in pull request #1154: [SeaTunnel #990][FOLLOWUP] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
simon824 commented on a change in pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154#discussion_r790666391
##########
File path: seatunnel-connectors/seatunnel-connector-spark-doris/src/main/scala/org/apache/seatunnel/spark/sink/Doris.scala
##########
@@ -68,35 +69,23 @@ class Doris extends SparkBatchSink with Serializable {
}
override def checkConfig(): CheckResult = {
- val checkResult =
- checkAllExists(config, Config.HOST, Config.DATABASE, Config.TABLE_NAME, Config.USER, Config.PASSWORD)
+ checkAllExists(config, HOST, DATABASE, TABLE_NAME, USER, PASSWORD)
+ }
- if (!checkResult.isSuccess) {
- checkResult
- } else if (config.hasPath(Config.USER) && !config.hasPath(Config.PASSWORD) || config.hasPath(
- Config.PASSWORD) && !config.hasPath(Config.USER)) {
Review comment:
This judgement is redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] simon824 commented on a change in pull request #1154: [SeaTunnel #990][FOLLOWUP] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
simon824 commented on a change in pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154#discussion_r790667357
##########
File path: seatunnel-connectors/seatunnel-connector-spark-doris/src/main/scala/org/apache/seatunnel/spark/sink/Doris.scala
##########
@@ -68,35 +69,23 @@ class Doris extends SparkBatchSink with Serializable {
}
override def checkConfig(): CheckResult = {
- val checkResult =
- checkAllExists(config, Config.HOST, Config.DATABASE, Config.TABLE_NAME, Config.USER, Config.PASSWORD)
+ checkAllExists(config, HOST, DATABASE, TABLE_NAME, USER, PASSWORD)
+ }
- if (!checkResult.isSuccess) {
- checkResult
- } else if (config.hasPath(Config.USER) && !config.hasPath(Config.PASSWORD) || config.hasPath(
- Config.PASSWORD) && !config.hasPath(Config.USER)) {
- CheckResult.error(Config.CHECK_USER_ERROR)
- } else {
- val host: String = config.getString(Config.HOST)
- val dataBase: String = config.getString(Config.DATABASE)
- val tableName: String = config.getString(Config.TABLE_NAME)
- this.apiUrl = s"http://$host/api/$dataBase/$tableName/_stream_load"
- if (TypesafeConfigUtils.hasSubConfig(config, Config.ARGS_PREFIX)) {
- val properties = TypesafeConfigUtils.extractSubConfig(config, Config.ARGS_PREFIX, true)
- val iterator = properties.entrySet().iterator()
- while (iterator.hasNext) {
- val map = iterator.next()
- val split = map.getKey.split("\\.")
- if (split.size == 2) {
- propertiesMap.put(split(1), String.valueOf(map.getValue.unwrapped))
- }
Review comment:
it's better to move this to `prepare()` after `checkConfig()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] CalvinKirs merged pull request #1154: [Feature] [connector] Optimize doris connector checkConfig
Posted by GitBox <gi...@apache.org>.
CalvinKirs merged pull request #1154:
URL: https://github.com/apache/incubator-seatunnel/pull/1154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org