You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Reynold Xin <rx...@databricks.com> on 2016/04/05 03:16:05 UTC
Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Nezih,
Have you had a chance to figure out why this is happening?
On Tue, Mar 22, 2016 at 1:32 AM, james <yi...@gmail.com> wrote:
> I guess different workload cause diff result ?
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Posted by Nezih Yigitbasi <ny...@netflix.com.INVALID>.
Thanks Imran. I will give it a shot when I have some time.
Nezih
On Thu, Apr 14, 2016 at 9:25 AM Imran Rashid <ir...@cloudera.com> wrote:
> Hi Nezih,
>
> I just reported a somewhat similar issue, and I have a potential fix --
> SPARK-14560, looks like you are already watching it :). You can try out
> that patch, you have to explicitly enable the change in behavior with
> "spark.shuffle.spillAfterRead=true". Honestly, I don't think these issues
> are the same, as I've always seen that case lead to acquiring 0 bytes,
> while in your case you are requesting GBs and getting something pretty
> close, so my hunch is that it is different ... but might be worth a shot to
> see if it is the issue.
>
> Turning on debug logging for TaskMemoryManager might help track the root
> cause -- you'll get information on which consumers are using memory and
> when there are spill attempts. (Note that even if the patch I have for
> SPARK-14560 doesn't fix your issue, it might still make those debug logs a
> bit more clear, since it'll report memory used by Spillables.)
>
> Imran
>
> On Mon, Apr 4, 2016 at 10:52 PM, Nezih Yigitbasi <
> nyigitbasi@netflix.com.invalid> wrote:
>
>> Nope, I didn't have a chance to track the root cause, and IIRC we didn't
>> observe it when dyn. alloc. is off.
>>
>> On Mon, Apr 4, 2016 at 6:16 PM Reynold Xin <rx...@databricks.com> wrote:
>>
>>> BTW do you still see this when dynamic allocation is off?
>>>
>>> On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <rx...@databricks.com> wrote:
>>>
>>>> Nezih,
>>>>
>>>> Have you had a chance to figure out why this is happening?
>>>>
>>>>
>>>> On Tue, Mar 22, 2016 at 1:32 AM, james <yi...@gmail.com> wrote:
>>>>
>>>>> I guess different workload cause diff result ?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
>>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>
Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Posted by Imran Rashid <ir...@cloudera.com>.
Hi Nezih,
I just reported a somewhat similar issue, and I have a potential fix --
SPARK-14560, looks like you are already watching it :). You can try out
that patch, you have to explicitly enable the change in behavior with
"spark.shuffle.spillAfterRead=true". Honestly, I don't think these issues
are the same, as I've always seen that case lead to acquiring 0 bytes,
while in your case you are requesting GBs and getting something pretty
close, so my hunch is that it is different ... but might be worth a shot to
see if it is the issue.
Turning on debug logging for TaskMemoryManager might help track the root
cause -- you'll get information on which consumers are using memory and
when there are spill attempts. (Note that even if the patch I have for
SPARK-14560 doesn't fix your issue, it might still make those debug logs a
bit more clear, since it'll report memory used by Spillables.)
Imran
On Mon, Apr 4, 2016 at 10:52 PM, Nezih Yigitbasi <
nyigitbasi@netflix.com.invalid> wrote:
> Nope, I didn't have a chance to track the root cause, and IIRC we didn't
> observe it when dyn. alloc. is off.
>
> On Mon, Apr 4, 2016 at 6:16 PM Reynold Xin <rx...@databricks.com> wrote:
>
>> BTW do you still see this when dynamic allocation is off?
>>
>> On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <rx...@databricks.com> wrote:
>>
>>> Nezih,
>>>
>>> Have you had a chance to figure out why this is happening?
>>>
>>>
>>> On Tue, Mar 22, 2016 at 1:32 AM, james <yi...@gmail.com> wrote:
>>>
>>>> I guess different workload cause diff result ?
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>> Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>>
>>>
>>
Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Posted by Nezih Yigitbasi <ny...@netflix.com.INVALID>.
Nope, I didn't have a chance to track the root cause, and IIRC we didn't
observe it when dyn. alloc. is off.
On Mon, Apr 4, 2016 at 6:16 PM Reynold Xin <rx...@databricks.com> wrote:
> BTW do you still see this when dynamic allocation is off?
>
> On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Nezih,
>>
>> Have you had a chance to figure out why this is happening?
>>
>>
>> On Tue, Mar 22, 2016 at 1:32 AM, james <yi...@gmail.com> wrote:
>>
>>> I guess different workload cause diff result ?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
>>> Sent from the Apache Spark Developers List mailing list archive at
>>> Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
>
Re: java.lang.OutOfMemoryError: Unable to acquire bytes of memory
Posted by Reynold Xin <rx...@databricks.com>.
BTW do you still see this when dynamic allocation is off?
On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <rx...@databricks.com> wrote:
> Nezih,
>
> Have you had a chance to figure out why this is happening?
>
>
> On Tue, Mar 22, 2016 at 1:32 AM, james <yi...@gmail.com> wrote:
>
>> I guess different workload cause diff result ?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>