You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "László Bodor (Jira)" <ji...@apache.org> on 2022/10/23 08:38:00 UTC
[jira] [Assigned] (TEZ-4448) Cannot submit Tez job when dag size is exceeds `ipc.maximum.data.length` and S3A is the filesystem
[ https://issues.apache.org/jira/browse/TEZ-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor reassigned TEZ-4448:
---------------------------------
Assignee: Karel Kolman
> Cannot submit Tez job when dag size is exceeds `ipc.maximum.data.length` and S3A is the filesystem
> --------------------------------------------------------------------------------------------------
>
> Key: TEZ-4448
> URL: https://issues.apache.org/jira/browse/TEZ-4448
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.9.2
> Reporter: Karel Kolman
> Assignee: Karel Kolman
> Priority: Major
> Labels: patch
> Attachments: TEZ-4448.20021007.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Submitting a Tez DAG with serialized size exceeding *ipc.maximum.data.length* in EMR/S3 environment results in a *FileNotFoundException*
> h5. Stacktrace
> {noformat}
> Caused by: java.io.FileNotFoundException: No such file or directory: s3://my-s3-bucket/my-job/.staging-a223d0fa-d315-4d84-9594-529c6ba34066/.tez/application_1655350046147_595990/tez-dag.pb1
> at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2310)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2204)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2143)
> at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
> at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:676)
> at org.apache.tez.client.TezClient.submitDAG(TezClient.java:588)
> ... 22 more
> {noformat}
> h5. Reproducing
> - set *ipc.maximum.data.length* to *64*1024* instead of the default value of *64*1024*1024*
> - attempt to submit a Tez job (EMR/S3)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)