You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by John Fang <xi...@alibaba-inc.com> on 2017/02/10 04:35:31 UTC
Driver hung and happend out of memory while writing to console progress bar
[Stage 172:==============================> (10328 + 93) / 16144]
[Stage 172:==============================> (10329 + 93) / 16144]
[Stage 172:==============================> (10330 + 93) / 16144]
[Stage 172:==============================> (10331 + 93) / 16144]
[Stage 172:==============================> (10333 + 92) / 16144]
[Stage 172:==============================> (10333 + 93) / 16144]
[Stage 172:==============================> (10333 + 94) / 16144]
[Stage 172:==============================> (10334 + 94) / 16144]
[Stage 172:==============================> (10338 + 93) / 16144]
[Stage 172:==============================> (10339 + 92) / 16144]
[Stage 172:==============================> (10340 + 93) / 16144]
[Stage 172:==============================> (10341 + 92) / 16144]
[Stage 172:==============================> (10341 + 93) / 16144]
[Stage 172:==============================> (10342 + 93) / 16144]
[Stage 172:==============================> (10343 + 93) / 16144]
[Stage 172:==============================> (10344 + 92) / 16144]
[Stage 172:==============================> (10345 + 92) / 16144]
[Stage 172:==============================> (10345 + 93) / 16144]
[Stage 172:==============================> (10346 + 93) / 16144]
[Stage 172:==============================> (10348 + 92) / 16144]
[Stage 172:==============================> (10348 + 93) / 16144]
[Stage 172:==============================> (10349 + 92) / 16144]
[Stage 172:==============================> (10349 + 93) / 16144]
[Stage 172:==============================> (10350 + 92) / 16144]
[Stage 172:==============================> (10352 + 92) / 16144]
[Stage 172:==============================> (10353 + 92) / 16144]
[Stage 172:==============================> (10354 + 92) / 16144]
[Stage 172:==============================> (10355 + 92) / 16144]
[Stage 172:==============================> (10356 + 92) / 16144]
[Stage 172:==============================> (10356 + 93) / 16144]
[Stage 172:==============================> (10357 + 92) / 16144]
[Stage 172:==============================> (10357 + 93) / 16144]
[Stage 172:==============================> (10358 + 92) / 16144]
[Stage 172:==============================> (10358 + 93) / 16144]
[Stage 172:==============================> (10359 + 92) / 16144]
[Stage 172:==============================> (10359 + 93) / 16144]
[Stage 172:==============================> (10359 + 94) / 16144]
[Stage 172:==============================> (10361 + 92) / 16144]
[Stage 172:==============================> (10361 + 93) / 16144]
[Stage 172:==============================> (10362 + 92) / 16144]
[Stage 172:==============================> (10362 + 93) / 16144]
[Stage 172:==============================> (10363 + 93) / 16144]
[Stage 172:==============================> (10364 + 92) / 16144]
[Stage 172:==============================> (10365 + 92) / 16144]
[Stage 172:==============================> (10365 + 93) / 16144]
[Stage 172:==============================> (10366 + 92) / 16144]
[Stage 172:==============================> (10366 + 93) / 16144]
[Stage 172:==============================> (10367 + 92) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]
[Stage 172:==============================> (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156)
at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124)
at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189)
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"
Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430)
at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101)
at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71)
at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Re: Driver hung and happend out of memory while writing to console
progress bar
Posted by Spark User <sp...@gmail.com>.
How much memory have you allocated to the driver? Driver stores some state
for tracking the task, stage and job history that you can see in the spark
console, it does take up a significant portion of the heap, anywhere from
200MB - 1G, depending no your map reduce steps.
Either way that is a good place to start by checking how much memory you
have allocated to the driver. If it is sufficient , like in the order of
2- 3G + at least, then you will have to take heap dumps of the driver
process periodically and find out what objects grow over time.
On Fri, Feb 10, 2017 at 9:34 AM, Ryan Blue <rb...@netflix.com.invalid>
wrote:
> This isn't related to the progress bar, it just happened while in that
> section of code. Something else is taking memory in the driver, usually a
> broadcast table or something else that requires a lot of memory and happens
> on the driver.
>
> You should check your driver memory settings and the query plan (if this
> was SparkSQL) for this stage to investigate further.
>
> rb
>
> On Thu, Feb 9, 2017 at 8:41 PM, John Fang <xi...@alibaba-inc.com>
> wrote:
>
>> the spark version is 2.1.0
>>
>> ------------------------------------------------------------------
>> 发件人：方孝健(玄弟) <xi...@alibaba-inc.com>
>> 发送时间：2017年2月10日(星期五) 12:35
>> 收件人：spark-dev <de...@spark.apache.org>; spark-user <us...@spark.apache.org>
>> 主 题：Driver hung and happend out of memory while writing to console
>> progress bar
>>
>> [Stage 172:==============================> (10328 + 93) / 16144][Stage 172:==============================> (10329 + 93) / 16144][Stage 172:==============================> (10330 + 93) / 16144][Stage 172:==============================> (10331 + 93) / 16144][Stage 172:==============================> (10333 + 92) / 16144][Stage 172:==============================> (10333 + 93) / 16144][Stage 172:==============================> (10333 + 94) / 16144][Stage 172:==============================> (10334 + 94) / 16144][Stage 172:==============================> (10338 + 93) / 16144][Stage 172:==============================> (10339 + 92) / 16144][Stage 172:==============================> (10340 + 93) / 16144][Stage 172:==============================> (10341 + 92) / 16144][Stage 172:==============================> (10341 + 93) / 16144][Stage 172:==============================> (10342 + 93) / 16144][Stage 172:==============================> (10343 + 93) / 16144][Stage 172:==============================> (10344 + 92) / 16144][Stage 172:==============================> (10345 + 92) / 16144][Stage 172:==============================> (10345 + 93) / 16144][Stage 172:==============================> (10346 + 93) / 16144][Stage 172:==============================> (10348 + 92) / 16144][Stage 172:==============================> (10348 + 93) / 16144][Stage 172:==============================> (10349 + 92) / 16144][Stage 172:==============================> (10349 + 93) / 16144][Stage 172:==============================> (10350 + 92) / 16144][Stage 172:==============================> (10352 + 92) / 16144][Stage 172:==============================> (10353 + 92) / 16144][Stage 172:==============================> (10354 + 92) / 16144][Stage 172:==============================> (10355 + 92) / 16144][Stage 172:==============================> (10356 + 92) / 16144][Stage 172:==============================> (10356 + 93) / 16144][Stage 172:==============================> (10357 + 92) / 16144][Stage 172:==============================> (10357 + 93) / 16144][Stage 172:==============================> (10358 + 92) / 16144][Stage 172:==============================> (10358 + 93) / 16144][Stage 172:==============================> (10359 + 92) / 16144][Stage 172:==============================> (10359 + 93) / 16144][Stage 172:==============================> (10359 + 94) / 16144][Stage 172:==============================> (10361 + 92) / 16144][Stage 172:==============================> (10361 + 93) / 16144][Stage 172:==============================> (10362 + 92) / 16144][Stage 172:==============================> (10362 + 93) / 16144][Stage 172:==============================> (10363 + 93) / 16144][Stage 172:==============================> (10364 + 92) / 16144][Stage 172:==============================> (10365 + 92) / 16144][Stage 172:==============================> (10365 + 93) / 16144][Stage 172:==============================> (10366 + 92) / 16144][Stage 172:==============================> (10366 + 93) / 16144][Stage 172:==============================> (10367 + 92) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringBuilder.toString(StringBuilder.java:407) at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430) at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101) at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71) at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
>>
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
Re: Driver hung and happend out of memory while writing to console
progress bar
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
This isn't related to the progress bar, it just happened while in that
section of code. Something else is taking memory in the driver, usually a
broadcast table or something else that requires a lot of memory and happens
on the driver.
You should check your driver memory settings and the query plan (if this
was SparkSQL) for this stage to investigate further.
rb
On Thu, Feb 9, 2017 at 8:41 PM, John Fang <xi...@alibaba-inc.com>
wrote:
> the spark version is 2.1.0
>
> ------------------------------------------------------------------
> 发件人：方孝健(玄弟) <xi...@alibaba-inc.com>
> 发送时间：2017年2月10日(星期五) 12:35
> 收件人：spark-dev <de...@spark.apache.org>; spark-user <us...@spark.apache.org>
> 主 题：Driver hung and happend out of memory while writing to console
> progress bar
>
> [Stage 172:==============================> (10328 + 93) / 16144][Stage 172:==============================> (10329 + 93) / 16144][Stage 172:==============================> (10330 + 93) / 16144][Stage 172:==============================> (10331 + 93) / 16144][Stage 172:==============================> (10333 + 92) / 16144][Stage 172:==============================> (10333 + 93) / 16144][Stage 172:==============================> (10333 + 94) / 16144][Stage 172:==============================> (10334 + 94) / 16144][Stage 172:==============================> (10338 + 93) / 16144][Stage 172:==============================> (10339 + 92) / 16144][Stage 172:==============================> (10340 + 93) / 16144][Stage 172:==============================> (10341 + 92) / 16144][Stage 172:==============================> (10341 + 93) / 16144][Stage 172:==============================> (10342 + 93) / 16144][Stage 172:==============================> (10343 + 93) / 16144][Stage 172:==============================> (10344 + 92) / 16144][Stage 172:==============================> (10345 + 92) / 16144][Stage 172:==============================> (10345 + 93) / 16144][Stage 172:==============================> (10346 + 93) / 16144][Stage 172:==============================> (10348 + 92) / 16144][Stage 172:==============================> (10348 + 93) / 16144][Stage 172:==============================> (10349 + 92) / 16144][Stage 172:==============================> (10349 + 93) / 16144][Stage 172:==============================> (10350 + 92) / 16144][Stage 172:==============================> (10352 + 92) / 16144][Stage 172:==============================> (10353 + 92) / 16144][Stage 172:==============================> (10354 + 92) / 16144][Stage 172:==============================> (10355 + 92) / 16144][Stage 172:==============================> (10356 + 92) / 16144][Stage 172:==============================> (10356 + 93) / 16144][Stage 172:==============================> (10357 + 92) / 16144][Stage 172:==============================> (10357 + 93) / 16144][Stage 172:==============================> (10358 + 92) / 16144][Stage 172:==============================> (10358 + 93) / 16144][Stage 172:==============================> (10359 + 92) / 16144][Stage 172:==============================> (10359 + 93) / 16144][Stage 172:==============================> (10359 + 94) / 16144][Stage 172:==============================> (10361 + 92) / 16144][Stage 172:==============================> (10361 + 93) / 16144][Stage 172:==============================> (10362 + 92) / 16144][Stage 172:==============================> (10362 + 93) / 16144][Stage 172:==============================> (10363 + 93) / 16144][Stage 172:==============================> (10364 + 92) / 16144][Stage 172:==============================> (10365 + 92) / 16144][Stage 172:==============================> (10365 + 93) / 16144][Stage 172:==============================> (10366 + 92) / 16144][Stage 172:==============================> (10366 + 93) / 16144][Stage 172:==============================> (10367 + 92) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringBuilder.toString(StringBuilder.java:407) at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430) at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101) at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71) at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
>
>
--
Ryan Blue
Software Engineer
Netflix
Re: Driver hung and happend out of memory while writing to console
progress bar
Posted by Ryan Blue <rb...@netflix.com.INVALID>.
This isn't related to the progress bar, it just happened while in that
section of code. Something else is taking memory in the driver, usually a
broadcast table or something else that requires a lot of memory and happens
on the driver.
You should check your driver memory settings and the query plan (if this
was SparkSQL) for this stage to investigate further.
rb
On Thu, Feb 9, 2017 at 8:41 PM, John Fang <xi...@alibaba-inc.com>
wrote:
> the spark version is 2.1.0
>
> ------------------------------------------------------------------
> 发件人：方孝健(玄弟) <xi...@alibaba-inc.com>
> 发送时间：2017年2月10日(星期五) 12:35
> 收件人：spark-dev <de...@spark.apache.org>; spark-user <us...@spark.apache.org>
> 主 题：Driver hung and happend out of memory while writing to console
> progress bar
>
> [Stage 172:==============================> (10328 + 93) / 16144][Stage 172:==============================> (10329 + 93) / 16144][Stage 172:==============================> (10330 + 93) / 16144][Stage 172:==============================> (10331 + 93) / 16144][Stage 172:==============================> (10333 + 92) / 16144][Stage 172:==============================> (10333 + 93) / 16144][Stage 172:==============================> (10333 + 94) / 16144][Stage 172:==============================> (10334 + 94) / 16144][Stage 172:==============================> (10338 + 93) / 16144][Stage 172:==============================> (10339 + 92) / 16144][Stage 172:==============================> (10340 + 93) / 16144][Stage 172:==============================> (10341 + 92) / 16144][Stage 172:==============================> (10341 + 93) / 16144][Stage 172:==============================> (10342 + 93) / 16144][Stage 172:==============================> (10343 + 93) / 16144][Stage 172:==============================> (10344 + 92) / 16144][Stage 172:==============================> (10345 + 92) / 16144][Stage 172:==============================> (10345 + 93) / 16144][Stage 172:==============================> (10346 + 93) / 16144][Stage 172:==============================> (10348 + 92) / 16144][Stage 172:==============================> (10348 + 93) / 16144][Stage 172:==============================> (10349 + 92) / 16144][Stage 172:==============================> (10349 + 93) / 16144][Stage 172:==============================> (10350 + 92) / 16144][Stage 172:==============================> (10352 + 92) / 16144][Stage 172:==============================> (10353 + 92) / 16144][Stage 172:==============================> (10354 + 92) / 16144][Stage 172:==============================> (10355 + 92) / 16144][Stage 172:==============================> (10356 + 92) / 16144][Stage 172:==============================> (10356 + 93) / 16144][Stage 172:==============================> (10357 + 92) / 16144][Stage 172:==============================> (10357 + 93) / 16144][Stage 172:==============================> (10358 + 92) / 16144][Stage 172:==============================> (10358 + 93) / 16144][Stage 172:==============================> (10359 + 92) / 16144][Stage 172:==============================> (10359 + 93) / 16144][Stage 172:==============================> (10359 + 94) / 16144][Stage 172:==============================> (10361 + 92) / 16144][Stage 172:==============================> (10361 + 93) / 16144][Stage 172:==============================> (10362 + 92) / 16144][Stage 172:==============================> (10362 + 93) / 16144][Stage 172:==============================> (10363 + 93) / 16144][Stage 172:==============================> (10364 + 92) / 16144][Stage 172:==============================> (10365 + 92) / 16144][Stage 172:==============================> (10365 + 93) / 16144][Stage 172:==============================> (10366 + 92) / 16144][Stage 172:==============================> (10366 + 93) / 16144][Stage 172:==============================> (10367 + 92) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringBuilder.toString(StringBuilder.java:407) at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430) at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101) at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71) at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
>
>
--
Ryan Blue
Software Engineer
Netflix
回复：Driver hung and happend out of memory while writing to console progress bar
Posted by John Fang <xi...@alibaba-inc.com>.
the spark version is 2.1.0
------------------------------------------------------------------发件人：方孝健(玄弟) <xi...@alibaba-inc.com>发送时间：2017年2月10日(星期五) 12:35收件人：spark-dev <de...@spark.apache.org>; spark-user <us...@spark.apache.org>主 题：Driver hung and happend out of memory while writing to console progress bar
[Stage 172:==============================> (10328 + 93) / 16144][Stage 172:==============================> (10329 + 93) / 16144][Stage 172:==============================> (10330 + 93) / 16144][Stage 172:==============================> (10331 + 93) / 16144][Stage 172:==============================> (10333 + 92) / 16144][Stage 172:==============================> (10333 + 93) / 16144][Stage 172:==============================> (10333 + 94) / 16144][Stage 172:==============================> (10334 + 94) / 16144][Stage 172:==============================> (10338 + 93) / 16144][Stage 172:==============================> (10339 + 92) / 16144][Stage 172:==============================> (10340 + 93) / 16144][Stage 172:==============================> (10341 + 92) / 16144][Stage 172:==============================> (10341 + 93) / 16144][Stage 172:==============================> (10342 + 93) / 16144][Stage 172:==============================> (10343 + 93) / 16144][Stage 172:==============================> (10344 + 92) / 16144][Stage 172:==============================> (10345 + 92) / 16144][Stage 172:==============================> (10345 + 93) / 16144][Stage 172:==============================> (10346 + 93) / 16144][Stage 172:==============================> (10348 + 92) / 16144][Stage 172:==============================> (10348 + 93) / 16144][Stage 172:==============================> (10349 + 92) / 16144][Stage 172:==============================> (10349 + 93) / 16144][Stage 172:==============================> (10350 + 92) / 16144][Stage 172:==============================> (10352 + 92) / 16144][Stage 172:==============================> (10353 + 92) / 16144][Stage 172:==============================> (10354 + 92) / 16144][Stage 172:==============================> (10355 + 92) / 16144][Stage 172:==============================> (10356 + 92) / 16144][Stage 172:==============================> (10356 + 93) / 16144][Stage 172:==============================> (10357 + 92) / 16144][Stage 172:==============================> (10357 + 93) / 16144][Stage 172:==============================> (10358 + 92) / 16144][Stage 172:==============================> (10358 + 93) / 16144][Stage 172:==============================> (10359 + 92) / 16144][Stage 172:==============================> (10359 + 93) / 16144][Stage 172:==============================> (10359 + 94) / 16144][Stage 172:==============================> (10361 + 92) / 16144][Stage 172:==============================> (10361 + 93) / 16144][Stage 172:==============================> (10362 + 92) / 16144][Stage 172:==============================> (10362 + 93) / 16144][Stage 172:==============================> (10363 + 93) / 16144][Stage 172:==============================> (10364 + 92) / 16144][Stage 172:==============================> (10365 + 92) / 16144][Stage 172:==============================> (10365 + 93) / 16144][Stage 172:==============================> (10366 + 92) / 16144][Stage 172:==============================> (10366 + 93) / 16144][Stage 172:==============================> (10367 + 92) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringBuilder.toString(StringBuilder.java:407) at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430) at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101) at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71) at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
回复：Driver hung and happend out of memory while writing to console progress bar
Posted by John Fang <xi...@alibaba-inc.com>.
the spark version is 2.1.0
------------------------------------------------------------------发件人：方孝健(玄弟) <xi...@alibaba-inc.com>发送时间：2017年2月10日(星期五) 12:35收件人：spark-dev <de...@spark.apache.org>; spark-user <us...@spark.apache.org>主 题：Driver hung and happend out of memory while writing to console progress bar
[Stage 172:==============================> (10328 + 93) / 16144][Stage 172:==============================> (10329 + 93) / 16144][Stage 172:==============================> (10330 + 93) / 16144][Stage 172:==============================> (10331 + 93) / 16144][Stage 172:==============================> (10333 + 92) / 16144][Stage 172:==============================> (10333 + 93) / 16144][Stage 172:==============================> (10333 + 94) / 16144][Stage 172:==============================> (10334 + 94) / 16144][Stage 172:==============================> (10338 + 93) / 16144][Stage 172:==============================> (10339 + 92) / 16144][Stage 172:==============================> (10340 + 93) / 16144][Stage 172:==============================> (10341 + 92) / 16144][Stage 172:==============================> (10341 + 93) / 16144][Stage 172:==============================> (10342 + 93) / 16144][Stage 172:==============================> (10343 + 93) / 16144][Stage 172:==============================> (10344 + 92) / 16144][Stage 172:==============================> (10345 + 92) / 16144][Stage 172:==============================> (10345 + 93) / 16144][Stage 172:==============================> (10346 + 93) / 16144][Stage 172:==============================> (10348 + 92) / 16144][Stage 172:==============================> (10348 + 93) / 16144][Stage 172:==============================> (10349 + 92) / 16144][Stage 172:==============================> (10349 + 93) / 16144][Stage 172:==============================> (10350 + 92) / 16144][Stage 172:==============================> (10352 + 92) / 16144][Stage 172:==============================> (10353 + 92) / 16144][Stage 172:==============================> (10354 + 92) / 16144][Stage 172:==============================> (10355 + 92) / 16144][Stage 172:==============================> (10356 + 92) / 16144][Stage 172:==============================> (10356 + 93) / 16144][Stage 172:==============================> (10357 + 92) / 16144][Stage 172:==============================> (10357 + 93) / 16144][Stage 172:==============================> (10358 + 92) / 16144][Stage 172:==============================> (10358 + 93) / 16144][Stage 172:==============================> (10359 + 92) / 16144][Stage 172:==============================> (10359 + 93) / 16144][Stage 172:==============================> (10359 + 94) / 16144][Stage 172:==============================> (10361 + 92) / 16144][Stage 172:==============================> (10361 + 93) / 16144][Stage 172:==============================> (10362 + 92) / 16144][Stage 172:==============================> (10362 + 93) / 16144][Stage 172:==============================> (10363 + 93) / 16144][Stage 172:==============================> (10364 + 92) / 16144][Stage 172:==============================> (10365 + 92) / 16144][Stage 172:==============================> (10365 + 93) / 16144][Stage 172:==============================> (10366 + 92) / 16144][Stage 172:==============================> (10366 + 93) / 16144][Stage 172:==============================> (10367 + 92) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144][Stage 172:==============================> (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189) at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at java.lang.StringBuilder.toString(StringBuilder.java:407) at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430) at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101) at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71) at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)