You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "dang-stripe (via GitHub)" <gi...@apache.org> on 2023/06/12 23:06:06 UTC
[GitHub] [pinot] dang-stripe opened a new issue, #10900: Add retries to mitigate transient issues when creating Kafka consumer
dang-stripe opened a new issue, #10900:
URL: https://github.com/apache/pinot/issues/10900
we've recently dealt with transient errors related to kafka dns resolution. this causes realtime segment transitions from OFFLINE -> CONSUMING to go immediately into ERROR state. we manually reset the segment to resolve the issue. it'd be ideal if we had retries implemented so the segment transition eventually succeeds.
```
[2023-06-01 15:04:09.010403] WARN [ClientUtils] [HelixTaskExecutor-message_handle_thread_8:25] Couldn't resolve server kafka-broker:9093 from bootstrap.servers as DNS resolution failed
[2023-06-01 15:04:09.010792] org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
[2023-06-01 15:04:09.011226] Caused by: org.apache.pinot.shaded.org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
```
https://github.com/apache/pinot/blob/207036950ad5fc13725ee464a2d0a09f26776638/pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/src/main/java/org/apache/pinot/plugin/stream/kafka20/KafkaPartitionLevelConnectionHandler.java#L64
cc @Jackie-Jiang @navina @jadami10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add retries to mitigate transient issues when creating Kafka consumer [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe closed issue #10900: Add retries to mitigate transient issues when creating Kafka consumer
URL: https://github.com/apache/pinot/issues/10900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] dang-stripe commented on issue #10900: Add retries to mitigate transient issues when creating Kafka consumer
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #10900:
URL: https://github.com/apache/pinot/issues/10900#issuecomment-1588227394
@Jackie-Jiang we noticed kafka consumer handles memory leaks already by wrapping the constructor code in a try-catch: https://github.com/apache/kafka/blob/2.8.1/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L817-L823 so probably doesn't need to be dealt with here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add retries to mitigate transient issues when creating Kafka consumer [pinot]
Posted by "dang-stripe (via GitHub)" <gi...@apache.org>.
dang-stripe commented on issue #10900:
URL: https://github.com/apache/pinot/issues/10900#issuecomment-1853288541
Fixed by https://github.com/apache/pinot/pull/11040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org