You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2017/01/09 13:26:58 UTC
[jira] [Comment Edited] (SPARK-19111) S3 Mesos history upload fails
silently if too large
[ https://issues.apache.org/jira/browse/SPARK-19111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15811767#comment-15811767 ]
Steve Loughran edited comment on SPARK-19111 at 1/9/17 1:26 PM:
----------------------------------------------------------------
What's happening here is that (a) S3n isn't uploading until the final close(), making it an O(data) operation. Maybe, that close() call is happening in something catching/swallowing exceptions. Or simply the close() is taking so long to upload that some other thread actually terminates the process.
# please switch to S3a; it's the one that's being maintained. S3n is being deliberately left alone to avoid breaking things in different ways than today.
# and move to Hadoop 2.7.x JARs, where S3a is ready to play with
S3A on Hadoop 2.7 has the ability to do incremental writes of blocks of data, the Fast Output Stream. This removes size limits and, because it is incrementally uploading, the only actions in close() are: upload the remaining data (guaranteed to be < one block); wait for all outstanding block uploads to complete, and then PUT the final commit-multipart document. It will need careful tuning to avoid running out of heap, as it buffers in RAM, and the defaults are a bit overoptimistic as to upload bandwidth. YMMV.
Hadoop 2.8 adds HADOOP-13560, and the BlockOutputStream successor to the FastOutputStream, which can buffer via /tmp or off-heap buffers, scales nicely and is tested to many, many GB. I'd grab that as soon as you can.
Irrespective of which upload option is used, you aren't going to see any history file until close() completes: that's how object stores work.
To close then: use S3a & Hadoop 2.7.x; the fast output stream may make this problem go away through incremental/background writes of the history.
was (Author: stevel@apache.org):
What's happening here is that (a) S3n isn't uploading until the final close(), making it an O(data) operation. Maybe, that close() call is happening in something catching/swallowing exceptions. Or simply the close() is taking so long to upload that some other thread actually terminates the process.
# please switch to S3a; it's the one that's being maintained. S3n is being deliberately left alone to avoid breaking things in different ways than today.
# and move to Hadoop 2.7.x JARs, where S3a is ready to play with
S3A on Hadoop 2.7 hasthe ability to do incremental writes of blocks of data, the Fast Output Stream. This removes size limits and, because it is incrementally uploading, the only actions in close() are: upload the remaining data (guaranteed to be < one block); wait for all outstanding block uploads to complete, and then PUT the final commit-multipart document. It will need careful to avoid running out of heap, as it buffers in RAM, and the defaults are a bit overoptimistic as to upload bandwidth. YMMV.
Hadoop 2.8 adds HADOOP-13560, and the BlockOutputStream successor to the FastOutputStream, which can buffer via /tmp or off-heap buffers, scales nicely and is tested to many, many GB. I'd grab that as soon as you can.
Irrespective of which upload option is used, you aren't going to see any history file until close() completes: that's how object stores work.
To close then: use S3a & Hadoop 2.7.x; the fast output stream may make this problem go away through incremental/background writes of the history.
> S3 Mesos history upload fails silently if too large
> ---------------------------------------------------
>
> Key: SPARK-19111
> URL: https://issues.apache.org/jira/browse/SPARK-19111
> Project: Spark
> Issue Type: Bug
> Components: EC2, Mesos, Spark Core
> Affects Versions: 2.0.0
> Reporter: Charles Allen
>
> {code}
> 2017-01-06T21:32:32,928 INFO [main] org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://REDACTED:4041
> 2017-01-06T21:32:32,938 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.jvmGCTime
> 2017-01-06T21:32:32,939 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.shuffle.read.localBlocksFetched
> 2017-01-06T21:32:32,939 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.resultSerializationTime
> 2017-01-06T21:32:32,939 ERROR [heartbeat-receiver-event-loop-thread] org.apache.spark.scheduler.LiveListenerBus - SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(
> 364,WrappedArray())
> 2017-01-06T21:32:32,939 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.resultSize
> 2017-01-06T21:32:32,939 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.peakExecutionMemory
> 2017-01-06T21:32:32,939 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.shuffle.read.fetchWaitTime
> 2017-01-06T21:32:32,939 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.memoryBytesSpilled
> 2017-01-06T21:32:32,940 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.shuffle.read.remoteBytesRead
> 2017-01-06T21:32:32,940 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.diskBytesSpilled
> 2017-01-06T21:32:32,940 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.shuffle.read.localBytesRead
> 2017-01-06T21:32:32,940 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.shuffle.read.recordsRead
> 2017-01-06T21:32:32,940 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.executorDeserializeTime
> 2017-01-06T21:32:32,940 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: output/bytes
> 2017-01-06T21:32:32,941 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.executorRunTime
> 2017-01-06T21:32:32,941 INFO [SparkListenerBus] com.metamx.starfire.spark.SparkDriver - emitting metric: internal.metrics.shuffle.read.remoteBlocksFetched
> 2017-01-06T21:32:32,943 INFO [main] org.apache.hadoop.fs.s3native.NativeS3FileSystem - OutputStream for key 'eventLogs/remnant/46bf8f87-6de6-4da8-9cba-5b2fecd0875e-1387.inprogress' closed. Now beginning upload
> 2017-01-06T21:32:32,963 ERROR [heartbeat-receiver-event-loop-thread] org.apache.spark.scheduler.LiveListenerBus - SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(905,WrappedArray())
> 2017-01-06T21:32:32,973 ERROR [heartbeat-receiver-event-loop-thread] org.apache.spark.scheduler.LiveListenerBus - SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(519,WrappedArray())
> 2017-01-06T21:32:32,988 ERROR [heartbeat-receiver-event-loop-thread] org.apache.spark.scheduler.LiveListenerBus - SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpdate(596,WrappedArray())
> {code}
> Running spark on mesos, some large jobs fail to upload to the history server storage!
> A successful sequence of events in the log that yield an upload are as follows:
> {code}
> 2017-01-06T19:14:32,925 INFO [main] org.apache.hadoop.fs.s3native.NativeS3FileSystem - OutputStream for key 'eventLogs/remnant/46bf8f87-6de6-4da8-9cba-5b2fecd0875e-1434.inprogress' writing to tempfile '/mnt/tmp/hadoop/output-2516573909248961808.tmp'
> 2017-01-06T21:59:14,789 INFO [main] org.apache.hadoop.fs.s3native.NativeS3FileSystem - OutputStream for key 'eventLogs/remnant/46bf8f87-6de6-4da8-9cba-5b2fecd0875e-1434.inprogress' closed. Now beginning upload
> 2017-01-06T21:59:44,679 INFO [main] org.apache.hadoop.fs.s3native.NativeS3FileSystem - OutputStream for key 'eventLogs/remnant/46bf8f87-6de6-4da8-9cba-5b2fecd0875e-1434.inprogress' upload complete
> {code}
> But large jobs do not ever get to the {{upload complete}} log message, and instead exit before completion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org