You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by SNEHASISH DUTTA <in...@gmail.com> on 2018/02/09 07:56:54 UTC
There is no space for new record
Hi ,
I am facing the following when running on EMR
Caused by: java.lang.IllegalStateException: There is no space for new record
at
org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:226)
at
org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132)
at
org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250)
I am using spark 2.2 , what spark configuration should be changed/modified
to get this resolved
Regards,
Snehasish
Re: There is no space for new record
Posted by SNEHASISH DUTTA <in...@gmail.com>.
Hi,
Will it be possible to overcome this with some spark configuration tweak ,
since EMR has spark version available only till 2.2.1
Regards,
Snehasish
On Tue, Feb 13, 2018 at 2:00 PM, Marco Gaido <ma...@gmail.com> wrote:
> You can check all the versions where the fix is available on the
> JIRA SPARK-23376. Anyway it will be available in the upcoming 2.3.0 release.
>
> Thanks.
>
> On 13 Feb 2018 9:09 a.m., "SNEHASISH DUTTA" <in...@gmail.com>
> wrote:
>
>> Hi,
>>
>> In which version of Spark will this fix be available ?
>> The deployment is on EMR
>>
>> Regards,
>> Snehasish
>>
>> On Fri, Feb 9, 2018 at 8:51 PM, Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> It should be fixed by https://github.com/apache/spark/pull/20561 soon.
>>>
>>> On Fri, Feb 9, 2018 at 6:16 PM, Wenchen Fan <cl...@gmail.com> wrote:
>>>
>>>> This has been reported before: http://apache-spark-de
>>>> velopers-list.1001551.n3.nabble.com/java-lang-IllegalStateEx
>>>> ception-There-is-no-space-for-new-record-tc20108.html
>>>>
>>>> I think we may have a real bug here, but we need a reproduce. Can you
>>>> provide one? thanks!
>>>>
>>>> On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA <
>>>> info.snehasish@gmail.com> wrote:
>>>>
>>>>> Hi ,
>>>>>
>>>>> I am facing the following when running on EMR
>>>>>
>>>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>>>> record
>>>>> at org.apache.spark.util.collecti
>>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem
>>>>> orySorter.java:226)
>>>>> at org.apache.spark.sql.execution
>>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132)
>>>>> at org.apache.spark.sql.execution
>>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor
>>>>> ter(UnsafeFixedWidthAggregationMap.java:250)
>>>>>
>>>>> I am using pyspark 2.2 , what spark configuration should be
>>>>> changed/modified to get this resolved
>>>>>
>>>>>
>>>>> Regards,
>>>>> Snehasish
>>>>>
>>>>>
>>>>> Regards,
>>>>> Snehasish
>>>>>
>>>>> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <
>>>>> info.snehasish@gmail.com> wrote:
>>>>>
>>>>>> Hi ,
>>>>>>
>>>>>> I am facing the following when running on EMR
>>>>>>
>>>>>> Caused by: java.lang.IllegalStateException: There is no space for
>>>>>> new record
>>>>>> at org.apache.spark.util.collecti
>>>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem
>>>>>> orySorter.java:226)
>>>>>> at org.apache.spark.sql.execution
>>>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132)
>>>>>> at org.apache.spark.sql.execution
>>>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor
>>>>>> ter(UnsafeFixedWidthAggregationMap.java:250)
>>>>>>
>>>>>> I am using spark 2.2 , what spark configuration should be
>>>>>> changed/modified to get this resolved
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Snehasish
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: There is no space for new record
Posted by Marco Gaido <ma...@gmail.com>.
You can check all the versions where the fix is available on the
JIRA SPARK-23376. Anyway it will be available in the upcoming 2.3.0 release.
Thanks.
On 13 Feb 2018 9:09 a.m., "SNEHASISH DUTTA" <in...@gmail.com>
wrote:
> Hi,
>
> In which version of Spark will this fix be available ?
> The deployment is on EMR
>
> Regards,
> Snehasish
>
> On Fri, Feb 9, 2018 at 8:51 PM, Wenchen Fan <cl...@gmail.com> wrote:
>
>> It should be fixed by https://github.com/apache/spark/pull/20561 soon.
>>
>> On Fri, Feb 9, 2018 at 6:16 PM, Wenchen Fan <cl...@gmail.com> wrote:
>>
>>> This has been reported before: http://apache-spark-de
>>> velopers-list.1001551.n3.nabble.com/java-lang-IllegalStateEx
>>> ception-There-is-no-space-for-new-record-tc20108.html
>>>
>>> I think we may have a real bug here, but we need a reproduce. Can you
>>> provide one? thanks!
>>>
>>> On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA <
>>> info.snehasish@gmail.com> wrote:
>>>
>>>> Hi ,
>>>>
>>>> I am facing the following when running on EMR
>>>>
>>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>>> record
>>>> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
>>>> orter.insertRecord(UnsafeInMemorySorter.java:226)
>>>> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
>>>> (UnsafeKVExternalSorter.java:132)
>>>> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
>>>> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregatio
>>>> nMap.java:250)
>>>>
>>>> I am using pyspark 2.2 , what spark configuration should be
>>>> changed/modified to get this resolved
>>>>
>>>>
>>>> Regards,
>>>> Snehasish
>>>>
>>>>
>>>> Regards,
>>>> Snehasish
>>>>
>>>> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <
>>>> info.snehasish@gmail.com> wrote:
>>>>
>>>>> Hi ,
>>>>>
>>>>> I am facing the following when running on EMR
>>>>>
>>>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>>>> record
>>>>> at org.apache.spark.util.collecti
>>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem
>>>>> orySorter.java:226)
>>>>> at org.apache.spark.sql.execution
>>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132)
>>>>> at org.apache.spark.sql.execution
>>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor
>>>>> ter(UnsafeFixedWidthAggregationMap.java:250)
>>>>>
>>>>> I am using spark 2.2 , what spark configuration should be
>>>>> changed/modified to get this resolved
>>>>>
>>>>>
>>>>> Regards,
>>>>> Snehasish
>>>>>
>>>>
>>>>
>>>
>>
>
Re: There is no space for new record
Posted by SNEHASISH DUTTA <in...@gmail.com>.
Hi,
In which version of Spark will this fix be available ?
The deployment is on EMR
Regards,
Snehasish
On Fri, Feb 9, 2018 at 8:51 PM, Wenchen Fan <cl...@gmail.com> wrote:
> It should be fixed by https://github.com/apache/spark/pull/20561 soon.
>
> On Fri, Feb 9, 2018 at 6:16 PM, Wenchen Fan <cl...@gmail.com> wrote:
>
>> This has been reported before: http://apache-spark-de
>> velopers-list.1001551.n3.nabble.com/java-lang-IllegalStateEx
>> ception-There-is-no-space-for-new-record-tc20108.html
>>
>> I think we may have a real bug here, but we need a reproduce. Can you
>> provide one? thanks!
>>
>> On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA <info.snehasish@gmail.com
>> > wrote:
>>
>>> Hi ,
>>>
>>> I am facing the following when running on EMR
>>>
>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>> record
>>> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
>>> orter.insertRecord(UnsafeInMemorySorter.java:226)
>>> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
>>> (UnsafeKVExternalSorter.java:132)
>>> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
>>> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregatio
>>> nMap.java:250)
>>>
>>> I am using pyspark 2.2 , what spark configuration should be
>>> changed/modified to get this resolved
>>>
>>>
>>> Regards,
>>> Snehasish
>>>
>>>
>>> Regards,
>>> Snehasish
>>>
>>> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <
>>> info.snehasish@gmail.com> wrote:
>>>
>>>> Hi ,
>>>>
>>>> I am facing the following when running on EMR
>>>>
>>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>>> record
>>>> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
>>>> orter.insertRecord(UnsafeInMemorySorter.java:226)
>>>> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
>>>> (UnsafeKVExternalSorter.java:132)
>>>> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
>>>> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregatio
>>>> nMap.java:250)
>>>>
>>>> I am using spark 2.2 , what spark configuration should be
>>>> changed/modified to get this resolved
>>>>
>>>>
>>>> Regards,
>>>> Snehasish
>>>>
>>>
>>>
>>
>
Re: There is no space for new record
Posted by Wenchen Fan <cl...@gmail.com>.
It should be fixed by https://github.com/apache/spark/pull/20561 soon.
On Fri, Feb 9, 2018 at 6:16 PM, Wenchen Fan <cl...@gmail.com> wrote:
> This has been reported before: http://apache-spark-
> developers-list.1001551.n3.nabble.com/java-lang-
> IllegalStateException-There-is-no-space-for-new-record-tc20108.html
>
> I think we may have a real bug here, but we need a reproduce. Can you
> provide one? thanks!
>
> On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA <in...@gmail.com>
> wrote:
>
>> Hi ,
>>
>> I am facing the following when running on EMR
>>
>> Caused by: java.lang.IllegalStateException: There is no space for new
>> record
>> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
>> orter.insertRecord(UnsafeInMemorySorter.java:226)
>> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
>> (UnsafeKVExternalSorter.java:132)
>> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
>> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregatio
>> nMap.java:250)
>>
>> I am using pyspark 2.2 , what spark configuration should be
>> changed/modified to get this resolved
>>
>>
>> Regards,
>> Snehasish
>>
>>
>> Regards,
>> Snehasish
>>
>> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <info.snehasish@gmail.com
>> > wrote:
>>
>>> Hi ,
>>>
>>> I am facing the following when running on EMR
>>>
>>> Caused by: java.lang.IllegalStateException: There is no space for new
>>> record
>>> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
>>> orter.insertRecord(UnsafeInMemorySorter.java:226)
>>> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
>>> (UnsafeKVExternalSorter.java:132)
>>> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
>>> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregatio
>>> nMap.java:250)
>>>
>>> I am using spark 2.2 , what spark configuration should be
>>> changed/modified to get this resolved
>>>
>>>
>>> Regards,
>>> Snehasish
>>>
>>
>>
>
Re: There is no space for new record
Posted by Wenchen Fan <cl...@gmail.com>.
This has been reported before:
http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-IllegalStateException-There-is-no-space-for-new-record-tc20108.html
I think we may have a real bug here, but we need a reproduce. Can you
provide one? thanks!
On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA <in...@gmail.com>
wrote:
> Hi ,
>
> I am facing the following when running on EMR
>
> Caused by: java.lang.IllegalStateException: There is no space for new
> record
> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
> orter.insertRecord(UnsafeInMemorySorter.java:226)
> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
> (UnsafeKVExternalSorter.java:132)
> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250)
>
> I am using pyspark 2.2 , what spark configuration should be
> changed/modified to get this resolved
>
>
> Regards,
> Snehasish
>
>
> Regards,
> Snehasish
>
> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <in...@gmail.com>
> wrote:
>
>> Hi ,
>>
>> I am facing the following when running on EMR
>>
>> Caused by: java.lang.IllegalStateException: There is no space for new
>> record
>> at org.apache.spark.util.collection.unsafe.sort.UnsafeInMemoryS
>> orter.insertRecord(UnsafeInMemorySorter.java:226)
>> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>
>> (UnsafeKVExternalSorter.java:132)
>> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMa
>> p.destructAndCreateExternalSorter(UnsafeFixedWidthAggregatio
>> nMap.java:250)
>>
>> I am using spark 2.2 , what spark configuration should be
>> changed/modified to get this resolved
>>
>>
>> Regards,
>> Snehasish
>>
>
>
Re: There is no space for new record
Posted by SNEHASISH DUTTA <in...@gmail.com>.
Hi ,
I am facing the following when running on EMR
Caused by: java.lang.IllegalStateException: There is no space for new record
at org.apache.spark.util.collection.unsafe.sort.
UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:226)
at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(
UnsafeKVExternalSorter.java:132)
at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.
destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250)
I am using pyspark 2.2 , what spark configuration should be
changed/modified to get this resolved
Regards,
Snehasish
Regards,
Snehasish
On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <in...@gmail.com>
wrote:
> Hi ,
>
> I am facing the following when running on EMR
>
> Caused by: java.lang.IllegalStateException: There is no space for new
> record
> at org.apache.spark.util.collection.unsafe.sort.
> UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:226)
> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(
> UnsafeKVExternalSorter.java:132)
> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.
> destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250)
>
> I am using spark 2.2 , what spark configuration should be changed/modified
> to get this resolved
>
>
> Regards,
> Snehasish
>