You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/09/22 20:26:14 UTC
[GitHub] [druid] sohansamant8 opened a new issue #11734: Druid Kafka Indexing fails without any details
sohansamant8 opened a new issue #11734:
URL: https://github.com/apache/druid/issues/11734
Hi,
My Kafka ingestion task are failing without any error details. I submit a supervisor specs and within less then 12 seconds task goes from running to failing state. And it continues to create new tasks as well which also starts failing.
I have two middle managers pod running and 2 Historical nodes running. Currently 3 ingestion jobs are running fine, but if we start adding 4th one the task starts failing and supervisor goes unhealthy.
Druid version: 0.21.0
Deployed on GCP K8 engine.
Any help on this is highly appreciated.
Attached is the config details:
**brokers:**
# Optionally specify for running broker as Deployment
kind: StatefulSet
nodeType: "broker"
druid.port: {{ .Values.application.druid.port }}
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/broker"
podDisruptionBudgetSpec:
maxUnavailable: 1
replicas: {{ .Values.application.druid.brokers.replicaCount }}
readinessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /druid/broker/v1/readiness
port: 8088
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
- name: DRUID_XMS
value: 512m
- name: DRUID_XMX
value: 2048m
runtime.properties: |
druid.service=druid/broker
druid.broker.http.numConnections=20
druid.server.http.numThreads=5
druid.processing.buffer.sizeBytes=268435456
druid.processing.numMergeBuffers=2
druid.processing.numThreads=1
druid.sql.enable=true
extra.jvm.options: |-
-Xmx512M
-Xms512M
**coordinators:**
# Optionally specify for running coordinator as Deployment
kind: StatefulSet
nodeType: "coordinator"
druid.port: {{ .Values.application.druid.port }}
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/coordinator-overlord"
podDisruptionBudgetSpec:
maxUnavailable: 1
replicas: {{ .Values.application.druid.coordinators.replicaCount }}
livenessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
readinessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
- name: DRUID_XMS
value: 1g
- name: DRUID_XMX
value: 2048m
runtime.properties: |
druid.service=druid/coordinator
druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord
druid.indexer.queue.startDelay=PT30S
druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata
druid.indexer.runner.pendingTasksRunnerNumThreads=8
druid.coordinator.maxNumConcurrentSubTasks=5
extra.jvm.options: |-
-Xmx512M
-Xms512M
**historicals:**
kind: StatefulSet
nodeType: "historical"
druid.port: {{ .Values.application.druid.port }}
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/data/historical"
podDisruptionBudgetSpec:
maxUnavailable: 1
replicas: {{ .Values.application.druid.historicals.replicaCount }}
livenessProbe:
initialDelaySeconds: 1800
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
readinessProbe:
httpGet:
path: /druid/historical/v1/readiness
port: {{ .Values.application.druid.port }}
periodSeconds: 10
failureThreshold: 18
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
- name: DRUID_XMS
value: 1500m
- name: DRUID_XMX
value: 1500m
runtime.properties: |
druid.service=druid/historical
druid.server.http.numThreads=10
druid.processing.buffer.sizeBytes=536870912
druid.processing.numMergeBuffers=2
druid.processing.numThreads=2
druid.segmentCache.locations=[{\"path\":\"/druid/data/segments\",\"maxSize\":40000000000}]
druid.server.maxSize=40000000000
extra.jvm.options: |-
-Xmx512M
-Xms512M
**middlemanagers:**
druid.port: {{ .Values.application.druid.port }}
kind: StatefulSet
nodeType: middleManager
nodeConfigMountPath: /opt/druid/conf/druid/cluster/data/middleManager
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /var/secrets/google/key.json
- name: DRUID_XMX
value: 2048m
- name: DRUID_XMS
value: 2048m
resources:
requests:
memory: 2.5Gi
limits:
memory: 3.4Gi
podDisruptionBudgetSpec:
maxUnavailable: 1
replicas: {{ .Values.application.druid.middlemanagers.replicaCount }}
livenessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
readinessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
runtime.properties: |
druid.service=druid/middleManager
druid.worker.capacity=6
druid.indexer.runner.javaOpts=-server -Xmx3g -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.restoreTasksOnRestart=true
druid.indexer.task.baseTaskDir=var/druid/task
druid.server.http.numThreads=8
druid.indexer.fork.property.druid.processing.numMergeBuffers=2
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=104857600
druid.indexer.fork.property.druid.processing.numThreads=2
routers:
kind: StatefulSet
nodeType: "router"
druid.port: {{ .Values.application.druid.port }}
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/query/router"
podDisruptionBudgetSpec:
maxUnavailable: 1
replicas: {{ .Values.application.druid.routers.replicaCount }}
livenessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
readinessProbe:
initialDelaySeconds: 60
periodSeconds: 5
failureThreshold: 3
httpGet:
path: /status/health
port: {{ .Values.application.druid.port }}
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: key.json
- name: DRUID_XMX
value: 4096m
- name: DRUID_XMS
value: 4096m
runtime.properties: |
druid.service=druid/router
druid.processing.numThreads=1
# HTTP proxy
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.server.http.numThreads=100
# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
# Management proxy to coordinator / overlord: required for unified web console.
druid.router.managementProxy.enabled=true
extra.jvm.options: |-
-Xmx512M
-Xms512M
[TaskError.txt](https://github.com/apache/druid/files/7213567/TaskError.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #11734: Kafka Indexing task fails without any details
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #11734:
URL: https://github.com/apache/druid/issues/11734#issuecomment-925900054
You might be running into memory constraints after adding the 4th task. Have you had a chance to analyze the failed task logs? They should contain some helpful information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org