You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/05/18 08:10:30 UTC
[GitHub] [incubator-kyuubi] cxzl25 opened a new pull request, #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
cxzl25 opened a new pull request, #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691
### _Why are the changes needed?_
close #2690
When `SparkProcessBuilder` is constructed, the commands are generated according to `conf` and are immutable, so the tags will not take effect if they are set later.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#discussion_r875827070
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala:
##########
@@ -88,20 +90,19 @@ class KyuubiApplicationManager extends AbstractService("KyuubiApplicationManager
object KyuubiApplicationManager {
private def setupSparkYarnTag(tag: String, conf: KyuubiConf): Unit = {
- val originalTag = conf.getOption("spark.yarn.tags").map(_ + ",").getOrElse("")
- val newTag = s"${originalTag}KYUUBI,$tag"
- conf.set("spark.yarn.tags", newTag)
+ val originalTag = conf.getOption(SparkProcessBuilder.TAG_KEY).map(_ + ",").getOrElse("")
+ val newTag = s"${originalTag}KYUUBI" + Some(tag).map("," + _).getOrElse("")
+ conf.set(SparkProcessBuilder.TAG_KEY, newTag)
}
private def setupSparkK8sTag(tag: String, conf: KyuubiConf): Unit = {
conf.set("spark.kubernetes.driver.label.kyuubi_unique_tag", tag)
}
private def setupFlinkK8sTag(tag: String, conf: KyuubiConf): Unit = {
- // TODO: yarn.tags or flink.yarn.tags, the mess of flink settings confuses me now.
- val originalTag = conf.getOption("yarn.tags").map(_ + ",")
- val newTag = s"${originalTag}KYUUBI,$tag"
- conf.set("yarn.tags", newTag)
+ val originalTag = conf.getOption(FlinkProcessBuilder.TAG_KEY).map(_ + ",")
+ val newTag = s"${originalTag}KYUUBI" + Some(tag).map("," + _).getOrElse("")
Review Comment:
is it a long time bug ? the type of `originalTag` is a Option not a String ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#issuecomment-1129903805
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#2691](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (70d0d7d) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/60b9f6bc2966db6a3a2b046fbd1cd566aeb9771a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (60b9f6b) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
> :exclamation: Current head 70d0d7d differs from pull request most recent head f4a830f. Consider uploading reports for the commit f4a830f to get more accurate results
```diff
@@ Coverage Diff @@
## master #2691 +/- ##
=========================================
Coverage 64.07% 64.07%
Complexity 275 275
=========================================
Files 407 407
Lines 19076 19074 -2
Branches 2568 2568
=========================================
Hits 12222 12222
+ Misses 5702 5699 -3
- Partials 1152 1153 +1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...kyuubi/engine/spark/SparkBatchProcessBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtCYXRjaFByb2Nlc3NCdWlsZGVyLnNjYWxh) | `96.66% <ø> (ø)` | |
| [...ache/kyuubi/engine/spark/SparkProcessBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvc3BhcmsvU3BhcmtQcm9jZXNzQnVpbGRlci5zY2FsYQ==) | `83.60% <ø> (-0.27%)` | :arrow_down: |
| [.../org/apache/kyuubi/operation/KyuubiOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uLnNjYWxh) | `68.00% <0.00%> (-1.34%)` | :arrow_down: |
| [...apache/kyuubi/session/KyuubiBatchSessionImpl.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXNzaW9uL0t5dXViaUJhdGNoU2Vzc2lvbkltcGwuc2NhbGE=) | `90.90% <0.00%> (-0.40%)` | :arrow_down: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `96.62% <0.00%> (-0.12%)` | :arrow_down: |
| [...rg/apache/kyuubi/engine/trino/TrinoSqlEngine.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZXh0ZXJuYWxzL2t5dXViaS10cmluby1lbmdpbmUvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9reXV1YmkvZW5naW5lL3RyaW5vL1RyaW5vU3FsRW5naW5lLnNjYWxh) | `59.64% <0.00%> (+7.01%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [60b9f6b...f4a830f](https://codecov.io/gh/apache/incubator-kyuubi/pull/2691?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#issuecomment-1129804731
yeah I think we can, the engineRefId should be bound to the engine builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] cxzl25 commented on a diff in pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on code in PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#discussion_r875804233
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilderSuite.scala:
##########
@@ -32,6 +32,7 @@ class FlinkProcessBuilderSuite extends KyuubiFunSuite {
.set(
ENGINE_FLINK_JAVA_OPTIONS,
"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005")
+ .set("yarn.tags", "KYUUBI")
Review Comment:
Because `FlinkProcessBuilde`r will write the `yarn.tags` configuration when generating commands, the previous test is broken.
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilderSuite.scala:
##########
@@ -32,6 +32,7 @@ class FlinkProcessBuilderSuite extends KyuubiFunSuite {
.set(
ENGINE_FLINK_JAVA_OPTIONS,
"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005")
+ .set("yarn.tags", "KYUUBI")
Review Comment:
Because `FlinkProcessBuilder` will write the `yarn.tags` configuration when generating commands, the previous test is broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#issuecomment-1129712472
is it possible to resolve te `// TODO: Better to do this inside ProcBuilder` ? it's confused we update the conf outside the builder. So the result of commands can be immuable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] cxzl25 commented on pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#issuecomment-1129793224
> is it possible to resolve te `// TODO: Better to do this inside ProcBuilder` ? it's confused we update the conf outside the builder. So the result of commands can be immuable.
So can we do this?
Add the parameter `engineRefId` when constructing `ProcessBuilder`, and call `tagApplication` when `ProcessBuilder` build commands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#discussion_r875747341
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilder.scala:
##########
@@ -37,7 +37,8 @@ import org.apache.kyuubi.operation.log.OperationLog
class FlinkProcessBuilder(
override val proxyUser: String,
override val conf: KyuubiConf,
- val extraEngineLog: Option[OperationLog] = None)
+ val extraEngineLog: Option[OperationLog] = None,
+ val engineRefId: String = "")
Review Comment:
the default value is only for simplify test code ? if so, we can add a new constructor with comments for that to be clear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a diff in pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on code in PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#discussion_r875795467
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/engine/flink/FlinkProcessBuilderSuite.scala:
##########
@@ -32,6 +32,7 @@ class FlinkProcessBuilderSuite extends KyuubiFunSuite {
.set(
ENGINE_FLINK_JAVA_OPTIONS,
"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005")
+ .set("yarn.tags", "KYUUBI")
Review Comment:
why add this config ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on pull request #2691: [KYUUBI #2690] Make ProcessBuilder.commands immutable
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#issuecomment-1130962150
thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] cxzl25 commented on a diff in pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on code in PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#discussion_r875735597
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala:
##########
@@ -98,10 +100,9 @@ object KyuubiApplicationManager {
}
private def setupFlinkK8sTag(tag: String, conf: KyuubiConf): Unit = {
- // TODO: yarn.tags or flink.yarn.tags, the mess of flink settings confuses me now.
Review Comment:
Check the Flink configuration documentation, there is no such configuration as `flink.yarn.tags`.
Although Flink supports converting `flink.yarn.xx` to `yarn.xx`, there is no `yarn.tags` configuration in yarn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] cxzl25 commented on a diff in pull request #2691: [KYUUBI #2690] SparkProcessBuilder commands mutable
Posted by GitBox <gi...@apache.org>.
cxzl25 commented on code in PR #2691:
URL: https://github.com/apache/incubator-kyuubi/pull/2691#discussion_r875910949
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala:
##########
@@ -88,20 +90,19 @@ class KyuubiApplicationManager extends AbstractService("KyuubiApplicationManager
object KyuubiApplicationManager {
private def setupSparkYarnTag(tag: String, conf: KyuubiConf): Unit = {
- val originalTag = conf.getOption("spark.yarn.tags").map(_ + ",").getOrElse("")
- val newTag = s"${originalTag}KYUUBI,$tag"
- conf.set("spark.yarn.tags", newTag)
+ val originalTag = conf.getOption(SparkProcessBuilder.TAG_KEY).map(_ + ",").getOrElse("")
+ val newTag = s"${originalTag}KYUUBI" + Some(tag).map("," + _).getOrElse("")
+ conf.set(SparkProcessBuilder.TAG_KEY, newTag)
}
private def setupSparkK8sTag(tag: String, conf: KyuubiConf): Unit = {
conf.set("spark.kubernetes.driver.label.kyuubi_unique_tag", tag)
}
private def setupFlinkK8sTag(tag: String, conf: KyuubiConf): Unit = {
- // TODO: yarn.tags or flink.yarn.tags, the mess of flink settings confuses me now.
- val originalTag = conf.getOption("yarn.tags").map(_ + ",")
- val newTag = s"${originalTag}KYUUBI,$tag"
- conf.set("yarn.tags", newTag)
+ val originalTag = conf.getOption(FlinkProcessBuilder.TAG_KEY).map(_ + ",")
+ val newTag = s"${originalTag}KYUUBI" + Some(tag).map("," + _).getOrElse("")
Review Comment:
It should be accidentally introduced by 2445, let me fix it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed pull request #2691: [KYUUBI #2690] Make ProcessBuilder.commands immutable
Posted by GitBox <gi...@apache.org>.
ulysses-you closed pull request #2691: [KYUUBI #2690] Make ProcessBuilder.commands immutable
URL: https://github.com/apache/incubator-kyuubi/pull/2691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org