You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "imperio-wxm (via GitHub)" <gi...@apache.org> on 2023/02/24 08:52:58 UTC
[GitHub] [kyuubi] imperio-wxm opened a new issue, #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
imperio-wxm opened a new issue, #4409:
URL: https://github.com/apache/kyuubi/issues/4409
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Deployed two kyuubi servers, provide a unified external rest api through load balancing.
I submitted a batch task and got the task status through `/api/v1/batches/{batchId}` polling.
From the log below, all requests are sent to a kyuubi server with same ip:
`2023-02-24 16:14:23.726 status=200, the normal request, jobState is running`
`2023-02-24 16:15:01.045 Invalid batchId`
`2023-02-24 16:15:01.046 status=404, the abnormal request`
`2023-02-24 16:18:14.848 status=200, the normal request, jobState is finished`
Repeated testing found that this phenomenon occurs at the end of the job.
```java
2023-02-24 16:14:23.726 INFO org.apache.kyuubi.server.http.authentication.AuthenticationAuditLogger: user=401(auth:BASIC) ip=xxx.xxx.7.74 proxyIp=null method=GET uri=/api/v1/batches/7c1a4b85-f396-4f79-a2b8-e63348d9c4d7 params=null protocol=HTTP/1.1 status=200
2023-02-24 16:15:01.045 ERROR org.apache.kyuubi.server.api.v1.BatchesResource: Invalid batchId: 7c1a4b85-f396-4f79-a2b8-e63348d9c4d7
2023-02-24 16:15:01.046 INFO org.apache.kyuubi.server.http.authentication.AuthenticationAuditLogger: user=401(auth:BASIC) ip=xxx.xxx.7.74 proxyIp=null method=GET uri=/api/v1/batches/7c1a4b85-f396-4f79-a2b8-e63348d9c4d7 params=null protocol=HTTP/1.1 status=404
2023-02-24 16:15:12.498 INFO org.apache.kyuubi.operation.BatchJobSubmission: Monitoring submitted SPARK batch[7c1a4b85-f396-4f79-a2b8-e63348d9c4d7] job: spark-080f7446308347ab84c1fb81f1713e33
2023-02-24 16:18:14.848 INFO org.apache.kyuubi.server.http.authentication.AuthenticationAuditLogger: user=401(auth:BASIC) ip=xxx.xxx.7.74 proxyIp=null method=GET uri=/api/v1/batches/7c1a4b85-f396-4f79-a2b8-e63348d9c4d7 params=null protocol=HTTP/1.1 status=200
```
### Affects Version(s)
branch-1.7
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] imperio-wxm commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "imperio-wxm (via GitHub)" <gi...@apache.org>.
imperio-wxm commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443284653
@pan3793 You mean kyuubi server needs an RDBMS as metadata storage? How to configure, I did not find relevant information in the document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443263033
You need an RDBMS, e.g. MySQL to store the batch job metadata, so all Kyuubi instances can know all batch jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443293716
also cc @turboFei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443204852
Please provide your `kyuubi-defauls.conf`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] imperio-wxm commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "imperio-wxm (via GitHub)" <gi...@apache.org>.
imperio-wxm commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443230435
@pan3793
- kyuubi-configmap.yaml
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: kyuubi-defaults
data:
kyuubi-defaults.conf: |
kyuubi.kubernetes.namespace=spark-operator
kyuubi.authentication=NONE
kyuubi.engine.share.level=USER
kyuubi.engine.connection.url.use.hostname=false
kyuubi.frontend.connection.url.use.hostname=false
kyuubi.operation.language=SCALA
kyuubi.session.engine.spark.main.resource=local:///opt/spark/jars/xxxxx.jar
kyuubi.ha.zookeeper.acl.enabled=true
kyuubi.ha.addresses=xxxx:2181
kyuubi.frontend.protocols=THRIFT_BINARY,REST
# spark
spark.master=k8s://https://xxxx
spark.submit.deployMode=cluster
# k8s
spark.kubernetes.container.image=xxxxx
spark.kubernetes.container.image.pullPolicy=Always
```
- kyuubi-deployment.yaml
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: kyuubi-servers
labels:
app: kyuubi-server
role: kyuubi-server
spec:
replicas: 2
selector:
matchLabels:
app: kyuubi-server
role: kyuubi-server
template:
metadata:
labels:
app: kyuubi-server
role: kyuubi-server
spec:
containers:
- name: kyuubi-server
image: xxxx
imagePullPolicy: Always
env:
- name: KYUUBI_JAVA_OPTS
value: -Dkyuubi.metrics.reporters=PROMETHEUS -Dkyuubi.metrics.prometheus.port=8585 -Dkyuubi.metrics.prometheus.path=/
ports:
- name: frontend-port
containerPort: 10009
protocol: TCP
volumeMounts:
- name: kyuubi-defaults
mountPath: /opt/kyuubi/conf
volumes:
- name: kyuubi-defaults
configMap:
name: kyuubi-defaults
```
Everything else is default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443291568
It's called metadata, the name is borrowed from Livy
https://kyuubi.readthedocs.io/en/master/deployment/settings.html#metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] imperio-wxm commented on issue #4409: [Bug] REST API GET /batches/{batchId} return inconsistent
Posted by "imperio-wxm (via GitHub)" <gi...@apache.org>.
imperio-wxm commented on issue #4409:
URL: https://github.com/apache/kyuubi/issues/4409#issuecomment-1443289835
But from the log, all requests are sent to one kyuubi server, and no request is made to another.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org