[jira] [Commented] (HADOOP-12979) IOE in S3a: ${hadoop.tmp.dir}/s3a not configured

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218067#comment-15218067 ] 

Steve Loughran commented on HADOOP-12979:
-----------------------------------------

full stack
{code}

Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:809)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:809)
  at scala.Option.foreach(Option.scala:257)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:809)
  ...
  Cause: java.io.IOException: ${hadoop.tmp.dir}/s3a not configured
  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:269)
  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:349)
  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:421)
  at org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:198)
  at org.apache.hadoop.fs.s3a.S3AOutputStream.<init>(S3AOutputStream.java:91)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:488)
  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:921)
  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:814)
  at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
  at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
{code}

> IOE in S3a:  ${hadoop.tmp.dir}/s3a not configured
> -------------------------------------------------
>
>                 Key: HADOOP-12979
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12979
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>
> Running some spark s3a tests trigger an NPE in Hadoop <=2/7l IOE in 2.8 saying "${hadoop.tmp.dir}/s3a not configured".
> That's correct: there is no configuration option on the conf called "${hadoop.tmp.dir}/s3a ". There may be one called {{hadoop.tmp.dir}}, however.
> Essentially s3a is sending the wrong config option down, if it can't find {{fs.s3a.buffer.dir}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)