[jira] [Commented] (SPARK-2671) BlockObjectWriter should create parent directory when the directory doesn't exist

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14073548#comment-14073548 ] 

Apache Spark commented on SPARK-2671:
-------------------------------------

User 'sarutak' has created a pull request for this issue:
https://github.com/apache/spark/pull/1580

> BlockObjectWriter should create parent directory when the directory doesn't exist
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-2671
>                 URL: https://issues.apache.org/jira/browse/SPARK-2671
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Kousuke Saruta
>            Priority: Minor
>
> BlockObjectWriter#open expects parent directory is present. 
> {code}
>   override def open(): BlockObjectWriter = {
>     fos = new FileOutputStream(file, true)
>     ts = new TimeTrackingOutputStream(fos)
>     channel = fos.getChannel()
>     lastValidPosition = initialPosition
>     bs = compressStream(new BufferedOutputStream(ts, bufferSize))
>     objOut = serializer.newInstance().serializeStream(bs)
>     initialized = true
>     this
>   }
> {code}
> Normally, the parent directory is created by DiskBlockManager#createLocalDirs but, just in case, BlockObjectWriter#open should check the existence of the directory and create the directory if the directory does not exist.
> I think, recoverable error should be recovered.



--
This message was sent by Atlassian JIRA
(v6.2#6252)