[GitHub] [iceberg] alex-shchetkov commented on issue #2991: No such file or directory when using multiple Spark executors + Iceberg in EMR

[GitBox <gi...@apache.org>]

alex-shchetkov commented on issue #2991:
URL: https://github.com/apache/iceberg/issues/2991#issuecomment-916134306


   I've ran into the same issue as @fcvr1010, where `-Djava.io.tmpdir=/tmp/driver/` did not help at first.
   However, after I kept re-trying different things on the same cluster, it suddenly worked. 
   Running in cluster mode, the driver node gets assigned somewhat randomly, but when it does, the `/tmp/driver` folder (as specified by `-Djava.io.tmpdir` finally gets created on that node. 
   Once it has been created on all nodes, the job started succeeding every time. 
   
   I can finally get the jobs to work on an always-on EMR cluster, but that's not ideal at all. I can't get it work right away on a brand new cluster.
   
   In my bootstrap script (that runs on all nodes during cluster provisioning) I added a `sudo mkdir /mnt/tmp/driver`, and while I no longer get `java.io.IOException`, the job simply hangs on the last stage every time, never completing. (Which is rather concerning. I'd rather the cluster fail and shut down, instead of wasting money by idling indefinitely)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org