You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yiannis Gkoufas <jo...@gmail.com> on 2015/02/17 18:00:22 UTC
Exception in Simple Job: Thread 'SortMerger spilling thread'
terminated due to an exception: The user-defined combiner failed in its
'open()' method
Hi there,
not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something
wrong.
I have this simple scala program:
val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
input.map(e => e.split(",")).filter(e => e.length>=4)
.map(e => (e(1),e(2),e(3).toDouble))
.groupBy(0,1)
.sum(2)
.writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")
env.execute("Test Flink")
And I get the following error:
org.apache.flink.client.program.ProgramInvocationException: The
program execution failed: java.lang.Exception: The data preparation
for task 'Reduce (SUM(2))' , caused an error: Error obtaining the
sorted input: Thread 'SortMerger spilling thread' terminated due to an
exception: The user-defined combiner failed in its 'open()' method.
at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted
input: Thread 'SortMerger spilling thread' terminated due to an
exception: The user-defined combiner failed in its 'open()' method.
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: The user-defined combiner failed in
its 'open()' method.
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
Caused by: java.io.IOException: The user-defined combiner failed in
its 'open()' method.
at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
Caused by: java.lang.IllegalStateException: The runtime context has
not been initialized.
at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
... 1 more
at org.apache.flink.client.program.Client.run(Client.java:345)
at org.apache.flink.client.program.Client.run(Client.java:304)
at org.apache.flink.client.program.Client.run(Client.java:298)
at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
at com.gmp.MyJob$.main(MyJob.scala:33)
at com.gmp.MyJob.main(MyJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
at org.apache.flink.client.program.Client.run(Client.java:250)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
Any help would be really appreciated!
Thank you!
Re: Exception in Simple Job: Thread 'SortMerger spilling thread'
terminated due to an exception: The user-defined combiner failed in its
'open()' method
Posted by Yiannis Gkoufas <jo...@gmail.com>.
Hi Fabian,
Thanks for letting me know! I will just give it a shot tomorrow.
Regards
On 17 Feb 2015 18:02, "Fabian Hueske" <fh...@gmail.com> wrote:
> In case you don't want to wait, this is the fix:
>
>
> https://github.com/fhueske/flink/commit/a1dd1f4e0ff3f45485eb40db9d607467f55b58c5
>
> 2015-02-17 18:57 GMT+01:00 Fabian Hueske <fh...@gmail.com>:
>
>> Hi,
>>
>> you are doing everything correct.
>>
>> This is a bug in the Flink runtime.
>> I created a JIRA (https://issues.apache.org/jira/browse/FLINK-1574) and
>> will push a fix later this evening once all tests have passed.
>>
>> Thanks for reporting the issue!
>>
>> Cheers, Fabian
>>
>> 2015-02-17 18:00 GMT+01:00 Yiannis Gkoufas <jo...@gmail.com>:
>>
>>> Hi there,
>>>
>>> not sure if its a bug of 0.9-SNAPSHOT version, or me just doing
>>> something wrong.
>>> I have this simple scala program:
>>>
>>> val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
>>> input.map(e => e.split(",")).filter(e => e.length>=4)
>>> .map(e => (e(1),e(2),e(3).toDouble))
>>> .groupBy(0,1)
>>> .sum(2)
>>> .writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")
>>>
>>> env.execute("Test Flink")
>>>
>>>
>>> And I get the following error:
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
>>> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
>>> at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>>> at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
>>> at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
>>> at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
>>> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
>>> ... 3 more
>>> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
>>> Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
>>> at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
>>> Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
>>> at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
>>> at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
>>> at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
>>> at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
>>> ... 1 more
>>>
>>> at org.apache.flink.client.program.Client.run(Client.java:345)
>>> at org.apache.flink.client.program.Client.run(Client.java:304)
>>> at org.apache.flink.client.program.Client.run(Client.java:298)
>>> at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
>>> at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
>>> at com.gmp.MyJob$.main(MyJob.scala:33)
>>> at com.gmp.MyJob.main(MyJob.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:483)
>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>>> at org.apache.flink.client.program.Client.run(Client.java:250)
>>> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
>>>
>>> Any help would be really appreciated!
>>>
>>> Thank you!
>>>
>>
>>
>
Re: Exception in Simple Job: Thread 'SortMerger spilling thread'
terminated due to an exception: The user-defined combiner failed in its
'open()' method
Posted by Fabian Hueske <fh...@gmail.com>.
In case you don't want to wait, this is the fix:
https://github.com/fhueske/flink/commit/a1dd1f4e0ff3f45485eb40db9d607467f55b58c5
2015-02-17 18:57 GMT+01:00 Fabian Hueske <fh...@gmail.com>:
> Hi,
>
> you are doing everything correct.
>
> This is a bug in the Flink runtime.
> I created a JIRA (https://issues.apache.org/jira/browse/FLINK-1574) and
> will push a fix later this evening once all tests have passed.
>
> Thanks for reporting the issue!
>
> Cheers, Fabian
>
> 2015-02-17 18:00 GMT+01:00 Yiannis Gkoufas <jo...@gmail.com>:
>
>> Hi there,
>>
>> not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something
>> wrong.
>> I have this simple scala program:
>>
>> val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
>> input.map(e => e.split(",")).filter(e => e.length>=4)
>> .map(e => (e(1),e(2),e(3).toDouble))
>> .groupBy(0,1)
>> .sum(2)
>> .writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")
>>
>> env.execute("Test Flink")
>>
>>
>> And I get the following error:
>>
>> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
>> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
>> at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
>> at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
>> at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
>> at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
>> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
>> ... 3 more
>> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
>> Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
>> at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
>> Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
>> at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
>> at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
>> at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
>> at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
>> ... 1 more
>>
>> at org.apache.flink.client.program.Client.run(Client.java:345)
>> at org.apache.flink.client.program.Client.run(Client.java:304)
>> at org.apache.flink.client.program.Client.run(Client.java:298)
>> at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
>> at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
>> at com.gmp.MyJob$.main(MyJob.scala:33)
>> at com.gmp.MyJob.main(MyJob.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>> at org.apache.flink.client.program.Client.run(Client.java:250)
>> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
>> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
>>
>> Any help would be really appreciated!
>>
>> Thank you!
>>
>
>
Re: Exception in Simple Job: Thread 'SortMerger spilling thread'
terminated due to an exception: The user-defined combiner failed in its
'open()' method
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
you are doing everything correct.
This is a bug in the Flink runtime.
I created a JIRA (https://issues.apache.org/jira/browse/FLINK-1574) and
will push a fix later this evening once all tests have passed.
Thanks for reporting the issue!
Cheers, Fabian
2015-02-17 18:00 GMT+01:00 Yiannis Gkoufas <jo...@gmail.com>:
> Hi there,
>
> not sure if its a bug of 0.9-SNAPSHOT version, or me just doing something
> wrong.
> I have this simple scala program:
>
> val input = env.readTextFile("hdfs://my.host.com:54310/home/sampleRaw")
> input.map(e => e.split(",")).filter(e => e.length>=4)
> .map(e => (e(1),e(2),e(3).toDouble))
> .groupBy(0,1)
> .sum(2)
> .writeAsCsv("hdfs://my.host.com:54310/home/testFlink2.csv")
>
> env.execute("Test Flink")
>
>
> And I get the following error:
>
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed: java.lang.Exception: The data preparation for task 'Reduce (SUM(2))' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
> at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
> at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607)
> at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133)
> at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94)
> at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
> ... 3 more
> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to an exception: The user-defined combiner failed in its 'open()' method.
> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785)
> Caused by: java.io.IOException: The user-defined combiner failed in its 'open()' method.
> at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264)
> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781)
> Caused by: java.lang.IllegalStateException: The runtime context has not been initialized.
> at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49)
> at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261)
> at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33)
> at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261)
> ... 1 more
>
> at org.apache.flink.client.program.Client.run(Client.java:345)
> at org.apache.flink.client.program.Client.run(Client.java:304)
> at org.apache.flink.client.program.Client.run(Client.java:298)
> at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55)
> at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530)
> at com.gmp.MyJob$.main(MyJob.scala:33)
> at com.gmp.MyJob.main(MyJob.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> at org.apache.flink.client.program.Client.run(Client.java:250)
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347)
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115)
>
> Any help would be really appreciated!
>
> Thank you!
>