You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2014/11/03 01:55:33 UTC

[jira] [Created] (TEZ-1731) OnDiskMerger can end up clobbering files across tasks with LocalDiskFetch

Siddharth Seth created TEZ-1731:
-----------------------------------

             Summary: OnDiskMerger can end up clobbering files across tasks with LocalDiskFetch
                 Key: TEZ-1731
                 URL: https://issues.apache.org/jira/browse/TEZ-1731
             Project: Apache Tez
          Issue Type: Bug
            Reporter: Siddharth Seth
            Assignee: Siddharth Seth
            Priority: Critical


When an on disk fetch starts with LOCAL files (optimize.local.fetch), the filename used by the merger is based on the source file name. This name can be the same for all tasks reading the same input on the node - and can result in files being overwritten between tasks, depending on the order in which events are processed, and the dir allocated by the local dir-allocator.

Leads to ChecksumExceptions, and FileNotFoundExceptions during the merge.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)