[jira] [Created] (TEZ-4448) Cannot submit Tez job when dag size is exceeds `ipc.maximum.data.length` and S3A is the filesystem

["Karel Kolman (Jira)" <ji...@apache.org>]

Karel Kolman created TEZ-4448:
---------------------------------

             Summary: Cannot submit Tez job when dag size is exceeds `ipc.maximum.data.length` and S3A is the filesystem
                 Key: TEZ-4448
                 URL: https://issues.apache.org/jira/browse/TEZ-4448
             Project: Apache Tez
          Issue Type: Improvement
    Affects Versions: 0.9.2
            Reporter: Karel Kolman


Submitting a Tez DAG with serialized size exceeding *ipc.maximum.data.length* in EMR/S3 environment results in a *FileNotFoundException*

h5. Stacktrace

{noformat}
Caused by: java.io.FileNotFoundException: No such file or directory: s3://my-s3-bucket/my-job/.staging-a223d0fa-d315-4d84-9594-529c6ba34066/.tez/application_1655350046147_595990/tez-dag.pb1
        at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2310)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
        at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
        at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
        at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:676)
        at org.apache.tez.client.TezClient.submitDAG(TezClient.java:588)
        ... 22 more
{noformat}

h5. Reproducing
- set *ipc.maximum.data.length* to *64*1024* instead of the default value of *64*1024*1024*
- attempt to submit a Tez job (EMR/S3)





--
This message was sent by Atlassian Jira
(v8.20.10#820010)