[GitHub] [hudi] gudladona commented on issue #6014: [SUPPORT] High runtime for a batch in SparkWriteHelper stage

[GitBox <gi...@apache.org>]

gudladona commented on issue #6014:
URL: https://github.com/apache/hudi/issues/6014#issuecomment-1235441748

   We faced a similar issue during this phase
   <img width="1916" alt="image" src="https://user-images.githubusercontent.com/7864088/188142313-b142a930-9f4b-4553-8229-cdf28bce1907.png">
   
   <img width="1908" alt="image" src="https://user-images.githubusercontent.com/7864088/188142431-239eca97-ff2f-493f-9c69-5579d481ece4.png">
   
   
   kafka fetch errors (at INFO level) are as follows:
   
   2022-09-02 06:26:12,606 INFO [Executor task launch worker for task 9.1 in stage 370.0 (TID 160203)] org.apache.kafka.clients.FetchSessionHandler:[Consumer clientId=consumer-spark-executor-hudi-ingest-auth-1, groupId=spark-executor-hudi-ingest-auth] Error sending fetch request (sessionId=1968849354, epoch=629) to node 8:
   org.apache.kafka.common.errors.DisconnectException
   
   and  warnings on the executors 
   
   2022-09-02 06:12:47,340 WARN [netty-rpc-env-timeout] org.apache.spark.rpc.netty.NettyRpcEnv:Ignored failure: java.util.concurrent.TimeoutException: Cannot receive any reply from ip-10-100-232-180.us-east-2.compute.internal:37403 in 10000 milliseconds
   2022-09-02 06:13:53,639 WARN [executor-heartbeater] org.apache.spark.executor.Executor:Issue communicating with driver in heartbeater
   org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10000 milliseconds]. This timeout is controlled by spark.executor.heartbeatInterval


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org