You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "ASiegeLion (via GitHub)" <gi...@apache.org> on 2023/09/26 03:44:53 UTC
[GitHub] [kyuubi] ASiegeLion opened a new issue, #5331: [Bug] Spark Engine should throw an exception to let K8s know when the engine fails to start.
ASiegeLion opened a new issue, #5331:
URL: https://github.com/apache/kyuubi/issues/5331
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
In the main method of SparkSQLEngine, exceptions are caught but not thrown. If the engine fails to start, the k8s status is displayed as finished, and the kyuubi server will not exit until the engine initialization times out.
### Affects Version(s)
master/1.7.1
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ASiegeLion commented on issue #5331: [Bug] Spark Engine should throw an exception to let K8s know when the engine fails to start.
Posted by "ASiegeLion (via GitHub)" <gi...@apache.org>.
ASiegeLion commented on issue #5331:
URL: https://github.com/apache/kyuubi/issues/5331#issuecomment-1735138285
> > YARN seems to be able to know
>
> If the engine does not throw an exception, the status seen in YARN is also SUCCESS.
>
> org.apache.spark.deploy.yarn.ApplicationMaster
>
> ```scala
> mainMethod.invoke(null, userArgs.toArray)
> finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
> ```
>
> https://github.com/apache/spark/blob/9a44dc4f9ee005f8e82f15bd731c9c870cc4a606/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L738-L739
>
> For example, setting a non-existent catalog leads to the initialization failure of the spark yarn cluster mode.
>
> ```ini
> spark.sql.catalog.spark_catalog=NOT_EXIST_CATALOG
> ```
You are right, the exception I see is the failure to start the ApplicationMaster, not the spark Engine. So I mistakenly thought that YARN seems to be able to know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [TASK][EASY][Bug] Spark Engine should throw an exception when the engine fails to start. [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #5331: [TASK][EASY][Bug] Spark Engine should throw an exception when the engine fails to start.
URL: https://github.com/apache/kyuubi/issues/5331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ASiegeLion commented on issue #5331: [Bug] Spark Engine should throw an exception to let K8s know when the engine fails to start.
Posted by "ASiegeLion (via GitHub)" <gi...@apache.org>.
ASiegeLion commented on issue #5331:
URL: https://github.com/apache/kyuubi/issues/5331#issuecomment-1735131614
> > YARN seems to be able to know
>
> If the engine does not throw an exception, the status seen in YARN is also SUCCESS.
>
> org.apache.spark.deploy.yarn.ApplicationMaster
>
> ```scala
> mainMethod.invoke(null, userArgs.toArray)
> finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
> ```
>
> https://github.com/apache/spark/blob/9a44dc4f9ee005f8e82f15bd731c9c870cc4a606/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L738-L739
>
> For example, setting a non-existent catalog leads to the initialization failure of the spark yarn cluster mode.
>
> ```ini
> spark.sql.catalog.spark_catalog=NOT_EXIST_CATALOG
> ```
You are right, the exception I saw was the failure to start ApplicationMaster, not spark Engine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] ASiegeLion commented on issue #5331: [Bug] Spark Engine should throw an exception to let K8s know when the engine fails to start.
Posted by "ASiegeLion (via GitHub)" <gi...@apache.org>.
ASiegeLion commented on issue #5331:
URL: https://github.com/apache/kyuubi/issues/5331#issuecomment-1734904485
> Thanks for reporting this issue, but seems it's not limited to K8s but a generic issue, right?
>
> PS: Please use text as much as possible, the picture is not searchable which makes the future explorer hard to find it.
If the createSpark method fails, YARN seems to be able to know, but K8S cannot. If the startEngine method fails, they don't know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #5331: [Bug] Spark Engine should throw an exception to let K8s know when the engine fails to start.
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #5331:
URL: https://github.com/apache/kyuubi/issues/5331#issuecomment-1734785977
Thanks for reporting this issue, but seems it's not limited to K8s but a generic issue, right?
PS: Please use text as much as possible, the picture is not searchable which makes the future explorer hard to find it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] cxzl25 commented on issue #5331: [Bug] Spark Engine should throw an exception to let K8s know when the engine fails to start.
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on issue #5331:
URL: https://github.com/apache/kyuubi/issues/5331#issuecomment-1735090878
> YARN seems to be able to know
If the engine does not throw an exception, the status seen in YARN is also SUCCESS.
org.apache.spark.deploy.yarn.ApplicationMaster
```scala
mainMethod.invoke(null, userArgs.toArray)
finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
```
https://github.com/apache/spark/blob/9a44dc4f9ee005f8e82f15bd731c9c870cc4a606/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L738-L739
For example, setting a non-existent catalog leads to the initialization failure of the spark yarn cluster mode.
```properties
spark.sql.catalog.spark_catalog=NOT_EXIST_CATALOG
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org