You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/10 17:21:00 UTC
[GitHub] [druid] ceastman-ibm opened a new issue #9496: druid management ui
in kubernetes fails if more than one coordinator/overlord process
ceastman-ibm opened a new issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496
### Affected Version
0.17.0
### Description
individual deployment/stateful set with seven replicas each of:
historical/data
coordinator/overlord
broker/query
ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "name" . }}
{{- with .Values.druid.ingress.annotations }}
annotations:
{{- . | toYaml | nindent 4 }}
{{- end }}
spec:
tls:
- hosts:
- {{ .Values.druid.ingress.hostnameRegional | quote }}
- {{ .Values.druid.ingress.hostnameGlobal | quote }}
secretName: {{ template "name" . }}-global-ssl-secret
rules:
- host: {{ .Values.druid.ingress.hostnameRegional }}
http:
paths:
- path: /
backend:
serviceName: druid-router
servicePort: 8888
- path: /druid/v2/sql
backend:
serviceName: druid-broker
servicePort: 8082
- path: /druid/coordinator
backend:
serviceName: druid-coordinator
servicePort: 8081
- path: /druid/indexer
backend:
serviceName: druid-overlord
servicePort: 8081
- host: {{ .Values.druid.ingress.hostnameGlobal }}
http:
paths:
- path: /
backend:
serviceName: druid-router
servicePort: 8888
- path: /druid/v2/sql
backend:
serviceName: druid-broker
servicePort: 8082
- path: /druid/coordinator
backend:
serviceName: druid-coordinator
servicePort: 8081
- path: /druid/indexer
backend:
serviceName: druid-overlord
servicePort: 8081
<img width="1680" alt="Screen Shot 2020-03-10 at 9 55 36 AM" src="https://user-images.githubusercontent.com/25593207/76340220-3a0b7780-62c9-11ea-8763-96ff6ce586d5.png">
when looking at the browser diagnostics i see these two urls are returning 404s:
9:57
https://<<redacted>>/druid/coordinator/v1/isLeader
https://<<redacted>>/druid/indexer/v1/isLeader
when i scale the deployment of the coordinator/overlord to one replica the issue goes away.
when i go directly to those two urls listed above it returns a json object:
isLeader: false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #9496: druid management
ui in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug edited a comment on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598355639
you can't point to service, druid(actually zookeeper client) needs to have all of zookeeper hosts individually listed like it is done in non-k8s deployments as well. (unless your zookeeper replica count is 1)
if you deployed zookeeper using a stateful set, make sure to add `serviceName` (as in https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster-zk.yaml#L29 ) in the StatefulSet. It is the magic attribute that sets `spec.subdomain` in launched pods which in turn leads to k8s cluster dns resolving fully qualified zk pods names e.g. "tiny-cluster-zk-0.tiny-cluster-zk, tiny-cluster-zk-1.tiny-cluster-zk .. etc"
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields doc describes same for pods.
Note: you might have to make your service a "headless service" but it might work with non-headless service as well so try that first, make sure "nslookup <fully qualified zk pod name>" works on all druid pods. See https://github.com/kubernetes/website/pull/16376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #9496: druid management
ui in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug edited a comment on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598355639
you can't point to service, druid(actually zookeeper client) needs to have all of zookeeper hosts individually listed like it is done in non-k8s deployments as well. (unless your zookeeper replica count is 1)
if you deployed zookeeper using a stateful set, make sure to add `serviceName` (as in https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster-zk.yaml#L29 ) in the StatefulSet is the magic attribute that sets `spec.subdomain` in launched pods which in turn leads to k8s cluster dns resolving fulling qualified zk pods names e.g. "tiny-cluster-zk-0.tiny-cluster-zk, tiny-cluster-zk-1.tiny-cluster-zk .. etc"
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields doc describes same for pods.
Note: you might have to make your service a "headless service" but it might work with non-headless service as well so try that first, make sure "nslookup <fully qualified zk pod name>" works on all druid pods. See https://github.com/kubernetes/website/pull/16376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #9496: druid management
ui in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug edited a comment on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598355639
you can't point to service, druid(actually zookeeper client) needs to have all of zookeeper hosts individually listed like it is done in non-k8s deployments as well.
if you deployed zookeeper using a stateful set, make sure to add `serviceName` (as in https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster-zk.yaml#L29 ) in the StatefulSet is the magic attribute that sets `spec.subdomain` in launched pods which in turn leads to k8s cluster dns resolving fulling qualified zk pods names e.g. "tiny-cluster-zk-0.tiny-cluster-zk, tiny-cluster-zk-1.tiny-cluster-zk .. etc"
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields doc describes same for pods.
Note: you might have to make your service a "headless service" but it might work with non-headless service as well so try that first, make sure "nslookup <fully qualified zk pod name>" works on all druid pods. See https://github.com/kubernetes/website/pull/16376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ceastman-ibm commented on issue #9496: druid management ui
in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
ceastman-ibm commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598843994
so i have them setup to run on the same kubernetes pods, i did 7 so that i could have at least a couple running in each data center that the kube cluster is spread across. its a multizone cluster running across three different data centers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ceastman-ibm commented on issue #9496: druid management ui
in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
ceastman-ibm commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598414828
@himanshug thank you for the information. i will give these things a try.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ceastman-ibm commented on issue #9496: druid management ui
in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
ceastman-ibm commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598343132
@himanshug currently i have pointing to a kubernetes service front ending a number of zookeeper pods:
./_common/common.runtime.properties:druid.zk.service.host=zookeeper
im not sure how having individual zookeepers listed would make those urls above work thou.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9496: druid management ui in
kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598299999
a wild guess: that sounds like something up with zookeeper. you need make sure all zookeeper replica pods are individually listed in all druid pods config in `druid.zk.service.hosts` property and make sure druid pods can reach all of the zk pods.
I use a zk sts that looks something like https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster-zk.yaml
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ceastman-ibm commented on issue #9496: druid management ui
in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
ceastman-ibm commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598783795
@himanshug that was it, it was my use of the zookeeper service as the zk host instead of listing out all the various headless zookeeper hostnames. i scaled the coordinator/overlord processes back to seven and all seems happy through the ui. thank you for your help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9496: druid management ui in
kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598355639
you can't point to service, druid(actually zookeeper client) needs to have all of zookeeper hosts individually listed like it is done in non-k8s deployments as well.
if you deployed zookeeper using a stateful set, make sure to add `serviceName` (as in https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster-zk.yaml#L29 ) in the StatefulSet is the magic attribute that sets `spec.subdomain` in launched pods which in turn leads to k8s cluster dns resolving fulling qualified zk pods names e.g. "tiny-cluster-zk-0.tiny-cluster-zk, tiny-cluster-zk-1.tiny-cluster-zk .. etc"
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields doc describes same for pods.
Note: you might have to make your service a "headless service" but it might work with non-headless service as well so try that first, make sure "nslookup <fully qualified zk pod name>" works on all druid pods. See https://github.com/kubernetes/website/pull/16376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ceastman-ibm closed issue #9496: druid management ui in
kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
ceastman-ibm closed issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9496: druid management ui in
kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598838233
glad to know that. however, 7 replicas for coordinator/overlord is overkill, you just want 2 replicas for each of them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #9496: druid management
ui in kubernetes fails if more than one coordinator/overlord process
Posted by GitBox <gi...@apache.org>.
himanshug edited a comment on issue #9496: druid management ui in kubernetes fails if more than one coordinator/overlord process
URL: https://github.com/apache/druid/issues/9496#issuecomment-598355639
you can't point to service, druid(actually zookeeper client) needs to have all of zookeeper hosts individually listed like it is done in non-k8s deployments as well. (unless your zookeeper replica count is 1)
if you deployed zookeeper using a stateful set, make sure to add `serviceName` (as in https://github.com/druid-io/druid-operator/blob/master/examples/tiny-cluster-zk.yaml#L29 ) in the StatefulSet. It is the magic attribute that sets `spec.subdomain` in launched pods which in turn leads to k8s cluster dns resolving fulling qualified zk pods names e.g. "tiny-cluster-zk-0.tiny-cluster-zk, tiny-cluster-zk-1.tiny-cluster-zk .. etc"
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-hostname-and-subdomain-fields doc describes same for pods.
Note: you might have to make your service a "headless service" but it might work with non-headless service as well so try that first, make sure "nslookup <fully qualified zk pod name>" works on all druid pods. See https://github.com/kubernetes/website/pull/16376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org