[jira] [Commented] (HADOOP-15634) LocalDirAllocator using up local nonDFS when set to S3

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558627#comment-16558627 ] 

Steve Loughran commented on HADOOP-15634:
-----------------------------------------

#. EMR team gets to field their problems, as they've forked the code
# That said: yes. LocalDirAllocator uses local directories. They are used in many places in Hadoop, often in different config options for different bits of the code
# S3 is not a filesystem. You can't write directly to it, so the s3 connectors all save data into blocks before upload. Where to? The local HDD via the LocalDirAllocator. S3 isnt going to work

I'm afraid you will have to accept that when code wants a local FS directory it means it and that you have to provide enough local storage for running applications.

That said: if Tez isn't cleaning up, that's an issue to take up with the EMR and Tez teams. For the latter, re-open this JIRA, move it to the Tez project. But come up with some evidence that they don't clean up first.

Closing as INVALID. Sorry



> LocalDirAllocator using up local nonDFS when set to S3
> ------------------------------------------------------
>
>                 Key: HADOOP-15634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.3
>         Environment: EMR-5.15, Hadoop-2.8.3, Hive-2.3.3, Tez-0.8.4, Beeline. 
> Target table is defined for ACID transactions with location on S3. 
> Insert source table is on S3. 
>            Reporter: Phani Kondapalli
>            Priority: Blocker
>
> Manually modified the yarn-site.xml from within the EMR, set the param yarn.nodemanager.local-dirs to point to s3, reloaded the services on Master and Core nodes. Disk seemed to stay intact but hdfs dfsadmin -report showed nonDFS usage and then finally it failed with below error.
> Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1532581073633_0001_2_00, diagnostics=[Task failed, taskId=task_1532581073633_0001_2_00_000898, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1532581073633_0001_2_00_000898_0:org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1532581073633_0001_2_00_000898_0_10013_1/file.out
>  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:441)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:151)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:132)
>  at org.apache.tez.runtime.library.common.task.local.output.TezTaskOutputFiles.getSpillFileForWrite(TezTaskOutputFiles.java:207)
>  at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.spill(PipelinedSorter.java:545)
> ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org