You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by on 2015/04/09 16:28:32 UTC
Jobs failing with KryoException (BufferOverflow)
My Spark (1.3.0) job is failing with
com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0,
required: 1+details
com.esotericsoftware.kryo.KryoException: Buffer overflow. Available:
0, required: 1
at com.esotericsoftware.kryo.io.Output.require(Output.java:138)
at com.esotericsoftware.kryo.io.Output.writeByte(Output.java:194)
at com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:599)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:566)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:318)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:293)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:161)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
--
Deepak
This is how am creating SparkContext (only once)
val conf = new SparkConf()
.setAppName(detail)
.set("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
.set("spark.kryoserializer.buffer.mb",arguments.get("buffersize").get)
.registerKryoClasses(Array(classOf[com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum]))
val sc = new SparkContext(conf)
Command:
./bin/spark-submit -v --master yarn-cluster --driver-class-path
/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar
--jars
/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar,/home/dvasthimal/spark1.3/spark_reporting_dep_only-1.0-SNAPSHOT.jar
--num-executors 100 --driver-memory 12g --driver-java-options
"-XX:MaxPermSize=6G" --executor-memory 12g --executor-cores 1 --queue
hdmi-express --class com.ebay.ep.poc.spark.reporting.SparkApp
/home/dvasthimal/spark1.3/spark_reporting-1.0-SNAPSHOT.jar
startDate=2015-04-6 endDate=2015-04-7
input=/user/dvasthimal/epdatasets_small/exptsession subcommand=viewItem
output=/user/dvasthimal/epdatasets/viewItem* buffersize=200*
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
buffer size is 200.
1. What is this buffer ?
2. What should be the value of this buffer ?
3. My Spark Job has many stages, does the above value need to be different
for each stage ?
Please clarify
Regards.
Deepak
Re: Jobs failing with KryoException (BufferOverflow)
Posted by .
Yeah, just ran with 2g for that setting and max.mb with 1068
I am trying to do a map-side join by using broadcast variable. This first
collects all the data (key, value) and then sends it. Its causing error
while running this stage.
On Thu, Apr 9, 2015 at 9:29 PM, Ted Yu <yu...@gmail.com> wrote:
> Typo in previous email, pardon me.
>
> Set "spark.driver.maxResultSize" to 1068 or higher.
>
> On Thu, Apr 9, 2015 at 8:57 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please set "spark.kryoserializer.buffer.max.mb" to 1068 (or higher).
>>
>> Cheers
>>
>> On Thu, Apr 9, 2015 at 8:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> Pressed send early.
>>>
>>> I had tried that with these settings
>>>
>>> buffersize=128 maxbuffersize=1024
>>>
>>> val conf = new SparkConf()
>>>
>>> .setAppName(detail)
>>>
>>> .set("spark.serializer",
>>> "org.apache.spark.serializer.KryoSerializer")
>>>
>>>
>>> .set("spark.kryoserializer.buffer.mb",arguments.get("buffersize").get)
>>>
>>>
>>> .set("spark.kryoserializer.buffer.max.mb",arguments.get("maxbuffersize").get)
>>>
>>>
>>> .registerKryoClasses(Array(classOf[com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum]))
>>>
>>>
>>> On Thu, Apr 9, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>>> wrote:
>>>
>>>> Yes i had tried that.
>>>>
>>>> Now i see this
>>>>
>>>> 15/04/09 07:58:08 INFO scheduler.DAGScheduler: Job 0 failed: collect at
>>>> VISummaryDataProvider.scala:38, took 275.334991 s
>>>> 15/04/09 07:58:08 ERROR yarn.ApplicationMaster: User class threw
>>>> exception: Job aborted due to stage failure: Total size of serialized
>>>> results of 4 tasks (1067.3 MB) is bigger than spark.driver.maxResultSize
>>>> (1024.0 MB)
>>>> org.apache.spark.SparkException: Job aborted due to stage failure:
>>>> Total size of serialized results of 4 tasks (1067.3 MB) is bigger than
>>>> spark.driver.maxResultSize (1024.0 MB)
>>>> at org.apache.spark.scheduler.DAGScheduler.org
>>>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>>>> at
>>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>>> at scala.Option.foreach(Option.scala:236)
>>>> at
>>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>>>> at
>>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>>>> at
>>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>>>> 15/04/09 07:58:08 INFO storage.BlockManagerInfo: Removed taskresult_4
>>>> on phxaishdc9dn0579.phx.ebay.com:42771 in memory (size: 273.5 MB,
>>>> free: 6.2 GB)
>>>> 15/04/09 07:58:08 INFO yarn.ApplicationMaster: Final app status:
>>>> FAILED, exitCode: 15, (reason: User
>>>>
>>>> On Thu, Apr 9, 2015 at 8:18 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> Please take a look at
>>>>> https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/Output.java?r=236
>>>>> , starting line 27.
>>>>>
>>>>> In Spark, you can control the maxBufferSize
>>>>> with "spark.kryoserializer.buffer.max.mb"
>>>>>
>>>>> Cheers
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Deepak
>>>>
>>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>
--
Deepak
Re: Jobs failing with KryoException (BufferOverflow)
Posted by Ted Yu <yu...@gmail.com>.
Typo in previous email, pardon me.
Set "spark.driver.maxResultSize" to 1068 or higher.
On Thu, Apr 9, 2015 at 8:57 AM, Ted Yu <yu...@gmail.com> wrote:
> Please set "spark.kryoserializer.buffer.max.mb" to 1068 (or higher).
>
> Cheers
>
> On Thu, Apr 9, 2015 at 8:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> Pressed send early.
>>
>> I had tried that with these settings
>>
>> buffersize=128 maxbuffersize=1024
>>
>> val conf = new SparkConf()
>>
>> .setAppName(detail)
>>
>> .set("spark.serializer",
>> "org.apache.spark.serializer.KryoSerializer")
>>
>>
>> .set("spark.kryoserializer.buffer.mb",arguments.get("buffersize").get)
>>
>>
>> .set("spark.kryoserializer.buffer.max.mb",arguments.get("maxbuffersize").get)
>>
>>
>> .registerKryoClasses(Array(classOf[com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum]))
>>
>>
>> On Thu, Apr 9, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com>
>> wrote:
>>
>>> Yes i had tried that.
>>>
>>> Now i see this
>>>
>>> 15/04/09 07:58:08 INFO scheduler.DAGScheduler: Job 0 failed: collect at
>>> VISummaryDataProvider.scala:38, took 275.334991 s
>>> 15/04/09 07:58:08 ERROR yarn.ApplicationMaster: User class threw
>>> exception: Job aborted due to stage failure: Total size of serialized
>>> results of 4 tasks (1067.3 MB) is bigger than spark.driver.maxResultSize
>>> (1024.0 MB)
>>> org.apache.spark.SparkException: Job aborted due to stage failure: Total
>>> size of serialized results of 4 tasks (1067.3 MB) is bigger than
>>> spark.driver.maxResultSize (1024.0 MB)
>>> at org.apache.spark.scheduler.DAGScheduler.org
>>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>>> at
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>>> at
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>>> at
>>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>>> at
>>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>>> at
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>> at
>>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>>> at scala.Option.foreach(Option.scala:236)
>>> at
>>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>>> at
>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>>> at
>>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>>> 15/04/09 07:58:08 INFO storage.BlockManagerInfo: Removed taskresult_4 on
>>> phxaishdc9dn0579.phx.ebay.com:42771 in memory (size: 273.5 MB, free:
>>> 6.2 GB)
>>> 15/04/09 07:58:08 INFO yarn.ApplicationMaster: Final app status: FAILED,
>>> exitCode: 15, (reason: User
>>>
>>> On Thu, Apr 9, 2015 at 8:18 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Please take a look at
>>>> https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/Output.java?r=236
>>>> , starting line 27.
>>>>
>>>> In Spark, you can control the maxBufferSize
>>>> with "spark.kryoserializer.buffer.max.mb"
>>>>
>>>> Cheers
>>>>
>>>
>>>
>>>
>>> --
>>> Deepak
>>>
>>>
>>
>>
>> --
>> Deepak
>>
>>
>
Re: Jobs failing with KryoException (BufferOverflow)
Posted by Ted Yu <yu...@gmail.com>.
Please set "spark.kryoserializer.buffer.max.mb" to 1068 (or higher).
Cheers
On Thu, Apr 9, 2015 at 8:54 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Pressed send early.
>
> I had tried that with these settings
>
> buffersize=128 maxbuffersize=1024
>
> val conf = new SparkConf()
>
> .setAppName(detail)
>
> .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
>
> .set("spark.kryoserializer.buffer.mb",arguments.get("buffersize").get)
>
>
> .set("spark.kryoserializer.buffer.max.mb",arguments.get("maxbuffersize").get)
>
>
> .registerKryoClasses(Array(classOf[com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum]))
>
>
> On Thu, Apr 9, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
>
>> Yes i had tried that.
>>
>> Now i see this
>>
>> 15/04/09 07:58:08 INFO scheduler.DAGScheduler: Job 0 failed: collect at
>> VISummaryDataProvider.scala:38, took 275.334991 s
>> 15/04/09 07:58:08 ERROR yarn.ApplicationMaster: User class threw
>> exception: Job aborted due to stage failure: Total size of serialized
>> results of 4 tasks (1067.3 MB) is bigger than spark.driver.maxResultSize
>> (1024.0 MB)
>> org.apache.spark.SparkException: Job aborted due to stage failure: Total
>> size of serialized results of 4 tasks (1067.3 MB) is bigger than
>> spark.driver.maxResultSize (1024.0 MB)
>> at org.apache.spark.scheduler.DAGScheduler.org
>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
>> at
>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>> at
>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>> at
>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
>> at scala.Option.foreach(Option.scala:236)
>> at
>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
>> at
>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
>> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
>> 15/04/09 07:58:08 INFO storage.BlockManagerInfo: Removed taskresult_4 on
>> phxaishdc9dn0579.phx.ebay.com:42771 in memory (size: 273.5 MB, free: 6.2
>> GB)
>> 15/04/09 07:58:08 INFO yarn.ApplicationMaster: Final app status: FAILED,
>> exitCode: 15, (reason: User
>>
>> On Thu, Apr 9, 2015 at 8:18 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Please take a look at
>>> https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/Output.java?r=236
>>> , starting line 27.
>>>
>>> In Spark, you can control the maxBufferSize
>>> with "spark.kryoserializer.buffer.max.mb"
>>>
>>> Cheers
>>>
>>
>>
>>
>> --
>> Deepak
>>
>>
>
>
> --
> Deepak
>
>
Re: Jobs failing with KryoException (BufferOverflow)
Posted by .
Pressed send early.
I had tried that with these settings
buffersize=128 maxbuffersize=1024
val conf = new SparkConf()
.setAppName(detail)
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryoserializer.buffer.mb",arguments.get("buffersize").get)
.set("spark.kryoserializer.buffer.max.mb",arguments.get("maxbuffersize").get)
.registerKryoClasses(Array(classOf[com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum]))
On Thu, Apr 9, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <de...@gmail.com> wrote:
> Yes i had tried that.
>
> Now i see this
>
> 15/04/09 07:58:08 INFO scheduler.DAGScheduler: Job 0 failed: collect at
> VISummaryDataProvider.scala:38, took 275.334991 s
> 15/04/09 07:58:08 ERROR yarn.ApplicationMaster: User class threw
> exception: Job aborted due to stage failure: Total size of serialized
> results of 4 tasks (1067.3 MB) is bigger than spark.driver.maxResultSize
> (1024.0 MB)
> org.apache.spark.SparkException: Job aborted due to stage failure: Total
> size of serialized results of 4 tasks (1067.3 MB) is bigger than
> spark.driver.maxResultSize (1024.0 MB)
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> 15/04/09 07:58:08 INFO storage.BlockManagerInfo: Removed taskresult_4 on
> phxaishdc9dn0579.phx.ebay.com:42771 in memory (size: 273.5 MB, free: 6.2
> GB)
> 15/04/09 07:58:08 INFO yarn.ApplicationMaster: Final app status: FAILED,
> exitCode: 15, (reason: User
>
> On Thu, Apr 9, 2015 at 8:18 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please take a look at
>> https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/Output.java?r=236
>> , starting line 27.
>>
>> In Spark, you can control the maxBufferSize
>> with "spark.kryoserializer.buffer.max.mb"
>>
>> Cheers
>>
>
>
>
> --
> Deepak
>
>
--
Deepak
Re: Jobs failing with KryoException (BufferOverflow)
Posted by .
Yes i had tried that.
Now i see this
15/04/09 07:58:08 INFO scheduler.DAGScheduler: Job 0 failed: collect at
VISummaryDataProvider.scala:38, took 275.334991 s
15/04/09 07:58:08 ERROR yarn.ApplicationMaster: User class threw exception:
Job aborted due to stage failure: Total size of serialized results of 4
tasks (1067.3 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
org.apache.spark.SparkException: Job aborted due to stage failure: Total
size of serialized results of 4 tasks (1067.3 MB) is bigger than
spark.driver.maxResultSize (1024.0 MB)
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
15/04/09 07:58:08 INFO storage.BlockManagerInfo: Removed taskresult_4 on
phxaishdc9dn0579.phx.ebay.com:42771 in memory (size: 273.5 MB, free: 6.2 GB)
15/04/09 07:58:08 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User
On Thu, Apr 9, 2015 at 8:18 PM, Ted Yu <yu...@gmail.com> wrote:
> Please take a look at
> https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/Output.java?r=236
> , starting line 27.
>
> In Spark, you can control the maxBufferSize
> with "spark.kryoserializer.buffer.max.mb"
>
> Cheers
>
--
Deepak
Re: Jobs failing with KryoException (BufferOverflow)
Posted by Ted Yu <yu...@gmail.com>.
Please take a look at
https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/Output.java?r=236
, starting line 27.
In Spark, you can control the maxBufferSize
with "spark.kryoserializer.buffer.max.mb"
Cheers