You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Devaraj K (JIRA)" <ji...@apache.org> on 2018/10/04 23:05:00 UTC
[jira] [Commented] (SPARK-25645) Add provision to disable
EventLoggingListener default flush/hsync/hflush for all events
[ https://issues.apache.org/jira/browse/SPARK-25645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639005#comment-16639005 ]
Devaraj K commented on SPARK-25645:
-----------------------------------
{code:java|title=Present Behavior(flushLogger=true for some events)|borderStyle=solid}
18/10/04 15:00:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/10/04 15:00:26 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/10/04 15:00:58 ERROR AsyncEventQueue: Dropping event from queue eventLog. This likely means one of the listeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
18/10/04 15:00:58 WARN AsyncEventQueue: Dropped 2 events from eventLog since Wed Dec 31 16:00:00 PST 1969.
18/10/04 15:01:58 WARN AsyncEventQueue: Dropped 216493 events from eventLog since Thu Oct 04 15:00:58 PDT 2018.
18/10/04 15:02:44 WARN AsyncEventQueue: [ADDED-LOG-BEFORE-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-appStatus, eventQueue.size(): 1
18/10/04 15:02:44 WARN AsyncEventQueue: [ADDED-LOG-AFTER-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-appStatus, eventQueue.size(): 0
18/10/04 15:02:44 WARN AsyncEventQueue: [ADDED-LOG-BEFORE-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-executorManagement, eventQueue.size(): 0
18/10/04 15:02:44 WARN AsyncEventQueue: [ADDED-LOG-AFTER-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-executorManagement, eventQueue.size(): 0
18/10/04 15:02:44 WARN AsyncEventQueue: [ADDED-LOG-BEFORE-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-eventLog, eventQueue.size(): 10000
18/10/04 15:03:39 WARN AsyncEventQueue: [ADDED-LOG-AFTER-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-eventLog, eventQueue.size(): 0
{code}
With the present behavior, taking 3 mins 14 sec to complete the application with the dropped events. And it is taking 55 sec to clear the eventLog queue at the end of the application.
{code:java|title=flush/hsync/hflush disabled for all events|borderStyle=solid}
18/10/04 14:51:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/10/04 14:51:34 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
18/10/04 14:53:54 WARN AsyncEventQueue: [ADDED-LOG-BEFORE-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-appStatus, eventQueue.size(): 0
18/10/04 14:53:54 WARN AsyncEventQueue: [ADDED-LOG-AFTER-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-appStatus, eventQueue.size(): 0
18/10/04 14:53:54 WARN AsyncEventQueue: [ADDED-LOG-BEFORE-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-executorManagement, eventQueue.size(): 0
18/10/04 14:53:54 WARN AsyncEventQueue: [ADDED-LOG-AFTER-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-executorManagement, eventQueue.size(): 0
18/10/04 14:53:54 WARN AsyncEventQueue: [ADDED-LOG-BEFORE-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-eventLog, eventQueue.size(): 0
18/10/04 14:53:54 WARN AsyncEventQueue: [ADDED-LOG-AFTER-DISPATCH-THREAD-JOIN] Thread.currentThread():main, dispatchThread:spark-listener-group-eventLog, eventQueue.size(): 0
{code}
With the disabled flush/hsync/hflush for all events, taking 2 mins 21 sec to complete the application without any dropped events. And also there are no pending events in eventLog queue at the end of the application.
> Add provision to disable EventLoggingListener default flush/hsync/hflush for all events
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-25645
> URL: https://issues.apache.org/jira/browse/SPARK-25645
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.2
> Reporter: Devaraj K
> Priority: Major
>
> {code:java|title=EventLoggingListener.scala|borderStyle=solid}
> private def logEvent(event: SparkListenerEvent, flushLogger: Boolean = false) {
> val eventJson = JsonProtocol.sparkEventToJson(event)
> // scalastyle:off println
> writer.foreach(_.println(compact(render(eventJson))))
> // scalastyle:on println
> if (flushLogger) {
> writer.foreach(_.flush())
> hadoopDataStream.foreach(ds => ds.getWrappedStream match {
> case wrapped: DFSOutputStream => wrapped.hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH))
> case _ => ds.hflush()
> })
> }
> {code}
> There are events which come with flushLogger=true and go through the underlying stream flush, Here I tried running apps with disabling the flush/hsync/hflush for all events and see that there is significant improvement in the app completion time and also there are no event drops, posting more details in the comments section.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org