Spark UI stdout/stderr links point to executors internal address

[Jhon Anderson Cardenas Diaz <jh...@gmail.com>]

*Environment:*

AWS EMR, yarn cluster.



*Description:*

On Spark ui, in Environment and Executors tabs, the links of stdout and
stderr point to the internal address of the executors. This would imply to
expose the executors so that links can be accessed. Shouldn't those links
be pointed to master then handled internally serving the master as a proxy
for these files instead of exposing the internal machines?



I have tried setting SPARK_PUBLIC_DNS and SPARK_LOCAL_IP variables so they
contain the master ip address. I also tried with this properties:
spark.yarn.appMasterEnv.SPARK_LOCAL_IP and
spark.yarn.appMasterEnv.SPARK_PUBLIC_DNS but it does not seem to work.


Any suggestion?
​