[I] [Workload] Connection refused error - Spark+Yarn [incubator-gluten]

["FelixYBW (via GitHub)" <gi...@apache.org>]

FelixYBW opened a new issue, #7612:
URL: https://github.com/apache/incubator-gluten/issues/7612

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   I met a strange issue when I test TPCDS workload. Recode here in case someone also meet. Maybe it's a well-known issue but I spent much time to debug it. It's not related to Gluten but run spark workload on Yarn.
   
   I have a master + 1 x worker. worker is configured as ip address. When yarn starts nodemanager on worker, it uses hostname instead of IP address. While the hostname is configured as 127.0.1.1 in /etc/hosts. Then the blockmanager is listening on 127.0.1.1:port. TPCH can pass without any issue, but TPCDS has lots of "connection refused" error, but the query can finish normally in spark-shell. In Pyspark it's hang.
   
   The solution is simple, delete the 127.0.1.1 line in /etc/hosts. I remember a configure in Yarn not to resolve hostname but use ip address directly, but I can't find it anymore.
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org

Re: [I] [Workload] Connection refused error - Spark+Yarn [incubator-gluten]

["FelixYBW (via GitHub)" <gi...@apache.org>]

FelixYBW closed issue #7612: [Workload] Connection refused error - Spark+Yarn
URL: https://github.com/apache/incubator-gluten/issues/7612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org