You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2018/12/20 15:49:00 UTC
[jira] [Commented] (SPARK-26284) Spark History server object vs
file storage behavior difference
[ https://issues.apache.org/jira/browse/SPARK-26284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725958#comment-16725958 ]
Steve Loughran commented on SPARK-26284:
----------------------------------------
This is how object stores work. PUTs / multipart puts of an object are atomic: they are only viaible on the close(). Flush is a no-op
The semantics of "flush()" are pretty vague, but those of {{Syncable.hflush()}} and {{Syncable.hsync()}} are strict: hsync: your changes must be visible to other readers; hflush() The changes must be persisted. The S3A output stream doesn't offer those guarantees, and so, when asked, [it says no|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java#L501]
This is going to have to be a WONTFIX. [Object stores are not filesystems|https://spark.apache.org/docs/2.3.1/cloud-integration.html#important-cloud-object-stores-are-not-real-filesystems]. Sorry.
> Spark History server object vs file storage behavior difference
> ---------------------------------------------------------------
>
> Key: SPARK-26284
> URL: https://issues.apache.org/jira/browse/SPARK-26284
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Damien Doucet-Girard
> Priority: Minor
>
> I am using the spark history server in order to view running/complete jobs on spark using the kubernetes scheduling backend introduced in 2.3.0. Using a local file path in both {color:#333333}{{spark.eventLog.dir}}{color} and {{spark.history.fs.logDirectory}}, I have no issue seeing both incomplete and completed tasks, with {{.inprogress}} files being flushed regularly. However, when using an {{s3a://}} path, it seems the calls to flush the file ([https://github.com/apache/spark/blob/dd518a196c2d40ae48034b8b0950d1c8045c02ed/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L152-L154)] don't actually upload the file to s3. Due to this, I am unable to see currently incomplete tasks using an s3a path.
> From the behavior I've observed, it only uploads on completion of the task (hadoop 2.7) or upon the log file filling up the block size set for s3a {{spark.hadoop.fs.s3a.multipart.size}} (hadoop 3.0.0). Is this intended behavior?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org