[jira] [Commented] (SPARK-17676) FsHistoryProvider should ignore hidden files

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524893#comment-15524893 ] 

Apache Spark commented on SPARK-17676:
--------------------------------------

User 'squito' has created a pull request for this issue:
https://github.com/apache/spark/pull/15250

> FsHistoryProvider should ignore hidden files
> --------------------------------------------
>
>                 Key: SPARK-17676
>                 URL: https://issues.apache.org/jira/browse/SPARK-17676
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Imran Rashid
>            Assignee: Imran Rashid
>            Priority: Minor
>
> FsHistoryProvider currently reads hidden files (beginning with ".") from the log dir.  However, it is writing a hidden file *itself* to that dir, which cannot be parsed, as part of a trick to find the scan time according to the file system:
> {code}
>     val fileName = "." + UUID.randomUUID().toString
>     val path = new Path(logDir, fileName)
>     val fos = fs.create(path)
> {code}
> It does delete the tmp file immediately, but we've seen cases where that race ends badly, and there is a logged error.  The error is harmless (the log file is ignored and spark moves on to the other log files), but the logged error is very confusing for users, so we should avoid it.
> {noformat}
> 2016-09-26 09:10:03,016 ERROR org.apache.spark.deploy.history.FsHistoryProvider: Exception encountered when attempting to load application log hdfs://XXX/user/spark/applicationHistory/.3a5e987c-ace5-4568-9ccd-6285010e399a 
> java.lang.IllegalArgumentException: Codec [3a5e987c-ace5-4568-9ccd-6285010e399a] is not available. Consider setting spark.io.compression.codec=lzf 
> at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:72) 
> at org.apache.spark.io.CompressionCodec$$anonfun$createCodec$1.apply(CompressionCodec.scala:72) 
> at scala.Option.getOrElse(Option.scala:120) 
> at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) 
> at org.apache.spark.scheduler.EventLoggingListener$$anonfun$8$$anonfun$apply$1.apply(EventLoggingListener.scala:309) 
> at org.apache.spark.scheduler.EventLoggingListener$$anonfun$8$$anonfun$apply$1.apply(EventLoggingListener.scala:309) 
> at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:189) 
> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:91) 
> at org.apache.spark.scheduler.EventLoggingListener$$anonfun$8.apply(EventLoggingListener.scala:309) 
> at org.apache.spark.scheduler.EventLoggingListener$$anonfun$8.apply(EventLoggingListener.scala:308) 
> at scala.Option.map(Option.scala:145) 
> at org.apache.spark.scheduler.EventLoggingListener$.openEventLog(EventLoggingListener.scala:308) 
> at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$replay(FsHistoryProvider.scala:518) 
> at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:359) 
> at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$10.apply(FsHistoryProvider.scala:356) 
> at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) 
> at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) 
> at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) 
> at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) 
> at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) 
> at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$mergeApplicationListing(FsHistoryProvider.scala:356)
> at org.apache.spark.deploy.history.FsHistoryProvider$$anonfun$checkForLogs$1$$anon$4.run(FsHistoryProvider.scala:277) 
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
> at java.lang.Thread.run(Thread.java:745)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org