You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zwangsheng (via GitHub)" <gi...@apache.org> on 2023/09/18 06:24:17 UTC
[GitHub] [kyuubi] zwangsheng opened a new issue, #5300: [Bug][V2] Face no local log exception when using v2 and rest batch submit
zwangsheng opened a new issue, #5300:
URL: https://github.com/apache/kyuubi/issues/5300
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Assume that two kyuubi server services exist and V2 rest version is enabled：
We submit job to kyuubi server, kyuubi server will make metadata and insert into db.
Kyuubi Servers have submitter(thread pool) to pick job for submitting.
Follow to the above logic:
1. kyuubi-1 submit this job into db, with `kyuubi.session.connection.url=kyuubi-1`
2. when kyuubi-0(maybe) pick up this job to do submit, will trigger session open, will update metadata with requestConf[using kyuubi-1 as connection url, this is wrong]
### Affects Version(s)
master
### Kyuubi Server Log Output
```logtalk
2023-09-18 13:34:16 [WARN] [KyuubiRestFrontendService-210] org.apache.kyuubi.server.api.RestExceptionMapper#82 - Error occurs on accessing REST API.
javax.ws.rs.NotFoundException: No local log found for batch: 11420abb-6b75-4846-ba93-d0f2e5ed9381
at org.apache.kyuubi.server.api.v1.BatchesResource.$anonfun$getBatchLocalLog$4(BatchesResource.scala:417) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at scala.Option.map(Option.scala:230) ~[scala-library-2.12.18.jar:?]
at org.apache.kyuubi.server.api.v1.BatchesResource.$anonfun$getBatchLocalLog$3(BatchesResource.scala:408) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at scala.Option.getOrElse(Option.scala:189) ~[scala-library-2.12.18.jar:?]
at org.apache.kyuubi.server.api.v1.BatchesResource.getBatchLocalLog(BatchesResource.scala:419) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_382]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_382]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.39.1.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.39.1.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.39.1.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.39.1.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.39.1.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.39.1.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[jersey-server-2.39.1.jar:?]
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.39.1.jar:?]
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.39.1.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ~[jersey-container-servlet-core-2.39.1.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ~[jersey-container-servlet-core-2.39.1.jar:?]
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.39.1.jar:?]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.apache.kyuubi.server.http.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.scala:157) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.server.http.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.scala:125) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.apache.kyuubi.server.http.authentication.KyuubiHttpAuthenticationFactory$HttpHandlerWrapperFactory$$anon$1$$anon$2.run(KyuubiHttpAuthenticationFactory.scala:64) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.server.http.authentication.KyuubiHttpAuthenticationFactory$HttpHandlerWrapperFactory$$anon$1$$anon$2.run(KyuubiHttpAuthenticationFactory.scala:62) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_382]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_382]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.kyuubi.server.http.authentication.KyuubiHttpAuthenticationFactory$HttpHandlerWrapperFactory$$anon$1.handle(KyuubiHttpAuthenticationFactory.scala:62) ~[kyuubi-server_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_382]
```
### Kyuubi Engine Log Output
```logtalk
2023-09-18 13:34:46 [ERROR] [main] org.apache.kyuubi.client.RestClient#189 - Error:
org.apache.http.client.HttpResponseException: status code: 404, reason phrase: {"message":"No local log found for batch: 11420abb-6b75-4846-ba93-d0f2e5ed9381"}
at org.apache.kyuubi.client.RestClient.lambda$doRequest$1(RestClient.java:177) ~[kyuubi-rest-client-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:223) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140) ~[httpclient-4.5.14.jar:4.5.14]
at org.apache.kyuubi.client.RestClient.doRequest(RestClient.java:181) ~[kyuubi-rest-client-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.client.RestClient.get(RestClient.java:80) ~[kyuubi-rest-client-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.client.RestClient.get(RestClient.java:74) ~[kyuubi-rest-client-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_382]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_382]
at org.apache.kyuubi.client.RetryableRestClient.invoke(RetryableRestClient.java:82) ~[kyuubi-rest-client-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at com.sun.proxy.$Proxy25.get(Unknown Source) ~[?:?]
at org.apache.kyuubi.client.BatchRestApi.getBatchLocalLog(BatchRestApi.java:101) ~[kyuubi-rest-client-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.log.LogBatchCommand.retrieveOperationLog$1(LogBatchCommand.scala:59) ~[kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.log.LogBatchCommand.$anonfun$doRun$3(LogBatchCommand.scala:68) ~[kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.log.LogBatchCommand.$anonfun$doRun$3$adapted(LogBatchCommand.scala:55) ~[kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiInstanceRestClient(RestClientFactory.scala:54) ~[kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.log.LogBatchCommand.$anonfun$doRun$1(LogBatchCommand.scala:55) ~[kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.log.LogBatchCommand.doRun(LogBatchCommand.scala:45) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.submit.SubmitBatchCommand.doRun(SubmitBatchCommand.scala:48) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.submit.SubmitBatchCommand.doRun(SubmitBatchCommand.scala:28) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala) [kyuubi-ctl_2.12-1.8.0.4-SNAPSHOT.jar:1.8.0.4-SNAPSHOT]
```
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed issue #5300: [Bug][V2] Face no local log exception when using v2 and rest batch submit
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #5300: [Bug][V2] Face no local log exception when using v2 and rest batch submit
URL: https://github.com/apache/kyuubi/issues/5300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org