You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "A-little-bit-of-data (via GitHub)" <gi...@apache.org> on 2024/07/16 07:24:28 UTC
[I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
A-little-bit-of-data opened a new issue, #6536:
URL: https://github.com/apache/kyuubi/issues/6536
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
When kyuubi-defaults.conf is modified and spark is started again, the new configuration is not loaded
When kyuubi-defaults.conf is configured as follows，The spark task started on k8s still reads the old configuration.
___trino___.spark.executor.instances=2
___trino___.spark.driver.cores=1
___trino___.spark.executor.cores=2
___trino___.spark.kubernetes.driver.limit.cores=2
___trino___.spark.kubernetes.executor.limit.cores=3
___trino___.spark.driver.memory=1g
___trino___.spark.executor.memory=2g
___trino___.spark.executor.instances=2
___trino___.spark.driver.cores=1
___trino___.spark.executor.cores=2
___trino___.spark.kubernetes.driver.limit.cores=2
___trino___.spark.kubernetes.executor.limit.cores=4
___trino___.spark.driver.memory=1g
___trino___.spark.executor.memory=4g
The spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL parameter is configured, but after completion, the spark driver pod still exists and the status is completed. It needs to be manually deleted.
### Affects Version(s)
1.9.1
### Kyuubi Server Log Output
```logtalk
/opt/kyuubi/externals/spark-3.5.1-bin-hadoop3/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
--conf spark.kyuubi.client.version=1.9.1 \
--conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/trino/kyuubi-spark-sql-engine.log.2 \
--conf spark.kyuubi.engine.share.level=USER \
--conf spark.kyuubi.engine.submit.time=1721112987920 \
--conf spark.kyuubi.engine.type=SPARK_SQL \
--conf spark.kyuubi.frontend.connection.url.use.hostname=false \
--conf spark.kyuubi.frontend.protocols=REST,THRIFT_BINARY \
--conf spark.kyuubi.ha.engine.ref.id=9653cc02-df96-4691-82b9-3452c5c2cdc4 \
--conf spark.kyuubi.ha.namespace=/kyuubi_1.9.1_USER_SPARK_SQL/trino/default \
--conf spark.kyuubi.ha.zookeeper.session.timeout=600000 \
--conf spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL \
--conf spark.kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M \
--conf spark.kyuubi.metrics.enabled=true \
--conf spark.kyuubi.metrics.reporters= \
--conf spark.kyuubi.session.engine.check.interval=PT1M \
--conf spark.kyuubi.session.engine.idle.timeout=PT2M \
--conf spark.kyuubi.session.real.user=trino \
--conf spark.app.name=trino \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=1g \
--conf spark.executor.cores=2 \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=2g \
--conf spark.kubernetes.driver.label.kyuubi-unique-tag=9653cc02-df96-4691-82b9-3452c5c2cdc4 \
--conf spark.kubernetes.driver.limit.cores=2 \
--conf spark.kubernetes.driver.pod.name=kyuubi-trino-9653cc02-df96-4691-82b9-3452c5c2cdc4-driver \
--conf spark.kubernetes.executor.limit.cores=3 \
--conf spark.kubernetes.executor.podNamePrefix=kyuubi-trino-9653cc02-df96-4691-82b9-3452c5c2cdc4 \
--conf spark.user=trino \
--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=trino \
--conf spark.executorEnv.SPARK_USER_NAME=trino \
--proxy-user trino /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.1.jar
/opt/kyuubi/externals/spark-3.5.1-bin-hadoop3/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
--conf spark.kyuubi.client.version=1.9.1 \
--conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/trino/kyuubi-spark-sql-engine.log.3 \
--conf spark.kyuubi.engine.share.level=USER \
--conf spark.kyuubi.engine.submit.time=1721114348590 \
--conf spark.kyuubi.engine.type=SPARK_SQL \
--conf spark.kyuubi.frontend.connection.url.use.hostname=false \
--conf spark.kyuubi.frontend.protocols=REST,THRIFT_BINARY \
--conf spark.kyuubi.ha.engine.ref.id=f93fdc9d-402b-4d9c-ba6e-7f01a4451f19 \
--conf spark.kyuubi.ha.namespace=/kyuubi_1.9.1_USER_SPARK_SQL/trino/default \
--conf spark.kyuubi.ha.zookeeper.session.timeout=600000 \
--conf spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL \
--conf spark.kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M \
--conf spark.kyuubi.metrics.enabled=true \
--conf spark.kyuubi.metrics.reporters= \
--conf spark.kyuubi.session.engine.check.interval=PT1M \
--conf spark.kyuubi.session.engine.idle.timeout=PT2M \
--conf spark.kyuubi.session.real.user=trino \
--conf spark.app.name=trino \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=1g \
--conf spark.executor.cores=2 \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=2g \
--conf spark.kubernetes.driver.label.kyuubi-unique-tag=f93fdc9d-402b-4d9c-ba6e-7f01a4451f19 \
--conf spark.kubernetes.driver.limit.cores=2 \
--conf spark.kubernetes.driver.pod.name=kyuubi-trino-f93fdc9d-402b-4d9c-ba6e-7f01a4451f19-driver \
--conf spark.kubernetes.executor.limit.cores=3 \
--conf spark.kubernetes.executor.podNamePrefix=kyuubi-trino-f93fdc9d-402b-4d9c-ba6e-7f01a4451f19 \
--conf spark.user=trino \
--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=trino \
--conf spark.executorEnv.SPARK_USER_NAME=trino \
--proxy-user trino /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.1.jar
```
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230289744
Kyuubi process consumes `kyuubi-defaults.conf` and `spark-submit` consumes `spark-defaults.conf`, it depends on your expectation to update and restart the corresponding configuration files and processes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230264553
> why does your configuration start with `trino.`?
It should be ___trino___. Used to configure multiple tenants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2232211608
correct.
in short, the modified `spark-defaults.conf` will take effect in the next `spark-submit`, so if you want to change **spark conf** without restarting Kyuubi process, just modify `spark-defaults.conf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230268160
> BTW, if you touch `kyuubi-defaults.conf`, you must restart the Kyuubi process. For your case, maybe you can modify `spark-defaults.conf` then the newly invoked `spark-submit` will see the updated Spark configurations.
My kyuubi is deployed on a container and controlled by configmap. Do I need to restart kyuubi?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230256555
> The `spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL` parameter is configured, but after completion, the spark driver pod still exists and the status is completed. It needs to be manually deleted.
it was reported by other users too, but can not be reproducible on my side. it's likely there are some issues on K8s client construction, you may want to enable trace level logs for Kyuubi server process to find what happened on your K8s client construction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230248626
why does your configuration start with `trino.`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230252724
BTW, if you touch `kyuubi-defaults.conf`, you must restart the Kyuubi process. For your case, maybe you can modify `spark-defaults.conf` then the newly invoked `spark-submit` will see the updated Spark configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2232540715
> the content of spark-defaults.conf before the modification is always read.
this is not expected, how do you make this assertion, can you provide some logs/screenshots or other proofs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data closed issue #6536: [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time
URL: https://github.com/apache/kyuubi/issues/6536
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2413036532
> after changing the spark-defaults.conf file, I waited for a while and then restarted it. You can see that the startup configuration is still the same.
`spark-submit` consumes `spark-defaults.conf` in the background and won't print the loaded configurations, all you see `spark-submit --conf abc=xyz` are loaded from `kyuubi-defaults.conf` or passed from the client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2232711203
> > the content of spark-defaults.conf before the modification is always read.
>
> this is not expected, how do you make this assertion, can you provide some logs/screenshots or other proofs?
There is a time in the initial submission log - conf spark.kyuubi.engine.submit.time. That is, after changing the spark-defaults.conf file, I waited for a while and then restarted it. You can see that the startup configuration is still the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2232501685
> correct.
>
> in short, the modified `spark-defaults.conf` will take effect in the next `spark-submit`, so if you want to change **spark conf** without restarting Kyuubi process, just modify `spark-defaults.conf`
But the current situation is that without restarting kyuubi, the content of spark-defaults.conf before the modification is always read. Only after restarting kyuubi can the modified content of spark-defaults.conf be read.
Here, after modifying spark-defaults.conf, I checked the spark-defaults.conf file in the kyuubi pod. The spark-defaults.conf file in the pod has been updated, but using spark-submit still reads the old configuration of the spark-defaults.conf file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2232814332
providing original logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2230285536
oh, the underscore is part of markdown syntax, you need to quote your inline code by backtick
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time [kyuubi]
Posted by "A-little-bit-of-data (via GitHub)" <gi...@apache.org>.
A-little-bit-of-data commented on issue #6536:
URL: https://github.com/apache/kyuubi/issues/6536#issuecomment-2232146941
> Kyuubi process consumes `kyuubi-defaults.conf` and `spark-submit` consumes `spark-defaults.conf`, it depends on your expectation to update and restart the corresponding configuration files and processes.
Sorry, I don't quite understand what you mean.
I understand that the spark on k8s task submitted by spark-submit is restarted after spark-defaults.conf is modified, but the new configuration is still not loaded. Under normal circumstances, I understand that I should have modified the spark-defaults.conf file under configmap kyuubi-spark, and the changes in the spark-defaults.conf file have been read in the kyuubi process. Then the spark on k8s task submitted by spark-submit again should load the spark-defaults.conf file I modified, instead of restarting the kyuubi application to read the modified content of the spark-defaults.conf file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org