[GitHub] [iceberg] kbendick edited a comment on issue #2991: No such file or directory when using multiple Spark executors + Iceberg in EMR

[GitBox <gi...@apache.org>]

kbendick edited a comment on issue #2991:
URL: https://github.com/apache/iceberg/issues/2991#issuecomment-901571240


   Hi @thaismonti1912, what Iceberg version are you using?
   
   Also, I have heard of this issue (vaguely), but it was EMR specific. In _that_ particular case, somebody was setting up their environment / cluster strangely. Of course, that doesn't mean that's the case here.
   
   This might be different though of course.
   
   > Configuration:
   > EMR version: emr-6.3.0
   > Master node: 1 instance r5.xlarge
   > Core node: 2 instances m5.xlarge
   > Spark version: 3.1.1
   
   Question:
   1) What Iceberg version are you using?
   2) Also, I noticed the iceberg jar is not listed in `--packages`. Do you have any EMR cluster set up scripts that install it? (To maybe deduce if there's a set up issue, though I don't know how those two would be related).
   
   > These are the scenarios:
   > 
   > spark application deployed in client mode on a cluster with one core node (works fine)
   > spark application deployed in cluster mode on a cluster with one core node (works fine)
   > spark application deployed in client mode on a cluster with multiple core nodes (works fine)
   > spark application deployed in cluster mode on a cluster with multiple core nodes (throws java.io.IOException: No such file or directory)
   
   Questions:
   1) When you say `1 core node` vs `multiple core nodes`, do you mean literally YARN core node or am I reading that incorrectly? It seems to be a driver vs executor issue, but `core node` has a very specific meaning, so I want to make sure we're on the same page :) 
   2) Does this happen _only_ when multiple executors are scheduled on the same node?
   3) Does this happen if you have 1 core node, many worker nodes, and several executors are allocated to the same node?
   
   If this is specific to only YARN (or I guess EMR) core nodes vs true worker nodes, this might a YARN specific behavior.
   
   If this is happening with multiple executors on just any given EMR node, then that's a different place to start investigating. If you can follow up on that clarification (as it seems you've done some substantial investigation already), that would be great.
   
   Given that we just use the normal `java.io.tmpdir`, this feels like either a YARN specific issue, an EMR specific issue, or a configuration issue. For example, I've run several executors using `S3FileIO` on the same node on Kubernetes (EKS and otherwise) without issue, and they've all used the default value of `java.io.tmpdir` (/tmp) - of course they're both virtualized in different ways, but still.
   
   And there are definitely many users that use 
   
   A few more questions:
   1) In your workaround, do you have to set a unique value for each executor / worker, or only the driver?
   2) If you run two programs at once, can you use the same `/tmp/driver` even if multiple drivers run on the same core node every time? 
   3) And in the example given above, do you have to create the `/tmp/driver/` folder ahead of time?
   
   > I found a workaround setting java.io.tmpdir property to a different directory in the driver as below:
   
   Based no the above, I'm guessing no.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org