You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2015/06/24 16:55:50 UTC

Datasets union CompilerException

Hi to all,

I'm running my flink-job on the new 0.9.0 version of Flink and now I get
this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException:
Bug: Plan generation for Unions picked a ship strategy between binary plan
operators.
at
org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
at
org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
at
org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
at
org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
at
org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at
org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
at
org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
at
org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at
org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at
org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at
org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at
org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
at
org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
at
org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,
Flavio

Re: Datasets union CompilerException

Posted by Stephan Ewen <se...@apache.org>.
I don't think it is related, but another bug...

On Tue, Jun 30, 2015 at 4:44 PM, Michele Bertoni <
michele1.bertoni@mail.polimi.it> wrote:

>  Hi yesterday on the union I faced an other problem:
> at runtime it was saying something like “Union cannot work with dataset of
> two different types” then it was showing the types and they were exactly
> the same (Tuple5<Long, String, Long, Long, MyClass>
> I solved it changing on field of the tuple from a custom object (MyClass)
> that was containing only a double to a native double and it worked
>
>  I don’t know if it is related but it may be
>
>  Michele
>
>
>  Il giorno 30/giu/2015, alle ore 10:57, Fabian Hueske <fh...@gmail.com>
> ha scritto:
>
>  Also, can you open a JIRA for the issue? Otherwise it might get lost on
> the mailing list.
>
>  Thanks you!
>
> 2015-06-30 10:56 GMT+02:00 Fabian Hueske <fh...@gmail.com>:
>
>>  Hi, is it possible to get a smaller version of that program that
>> reproduces the bug or give a few more details about the structure of the
>> job?
>>  Without any hints, it is very hard to reproduce and fix the bug.
>>
>> 2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <po...@okkam.it>:
>>
>>> Unfortunately not in public..moreover this error seems to be non
>>> deterministic..sometime the programs runs correctly :(
>>>
>>> On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
>>>
>>>> Hmm, that issue must have been introduced as part of a recent change.
>>>>
>>>> Can you share your program so we can reproduce abs fix the bug?
>>>>  Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <pompermaier@okkam.it
>>>> >:
>>>>
>>>>> Hi to all,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>   I'm running my flink-job on the new 0.9.0 version of Flink and now
>>>>> I get this exception:
>>>>>
>>>>>  Exception in thread "main"
>>>>> org.apache.flink.optimizer.CompilerException: Bug: Plan generation for
>>>>> Unions picked a ship strategy between binary plan operators.
>>>>> at
>>>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>>>>> at
>>>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>>>>> at
>>>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>>> at
>>>>> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>>>>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>>>>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>>>>> at
>>>>> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
>>>>> at
>>>>> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
>>>>> at
>>>>> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
>>>>>
>>>>>  How can I fix that?
>>>>>
>>>>>  Best,
>>>>> Flavio
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>

Re: Datasets union CompilerException

Posted by Michele Bertoni <mi...@mail.polimi.it>.
Hi yesterday on the union I faced an other problem:
at runtime it was saying something like “Union cannot work with dataset of two different types” then it was showing the types and they were exactly the same (Tuple5<Long, String, Long, Long, MyClass>
I solved it changing on field of the tuple from a custom object (MyClass) that was containing only a double to a native double and it worked

I don’t know if it is related but it may be

Michele


Il giorno 30/giu/2015, alle ore 10:57, Fabian Hueske <fh...@gmail.com>> ha scritto:

Also, can you open a JIRA for the issue? Otherwise it might get lost on the mailing list.

Thanks you!

2015-06-30 10:56 GMT+02:00 Fabian Hueske <fh...@gmail.com>>:
Hi, is it possible to get a smaller version of that program that reproduces the bug or give a few more details about the structure of the job?
Without any hints, it is very hard to reproduce and fix the bug.

2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <po...@okkam.it>>:
Unfortunately not in public..moreover this error seems to be non deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <se...@apache.org>> wrote:

Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?

Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <po...@okkam.it>>:
Hi to all,




I'm running my flink-job on the new 0.9.0 version of Flink and now I get this exception:

Exception in thread "main" org.apache.flink.optimizer.CompilerException: Bug: Plan generation for Unions picked a ship strategy between binary plan operators.
at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
at org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
at org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)

How can I fix that?

Best,
Flavio









Re: Datasets union CompilerException

Posted by Fabian Hueske <fh...@gmail.com>.
Also, can you open a JIRA for the issue? Otherwise it might get lost on the
mailing list.

Thanks you!

2015-06-30 10:56 GMT+02:00 Fabian Hueske <fh...@gmail.com>:

> Hi, is it possible to get a smaller version of that program that
> reproduces the bug or give a few more details about the structure of the
> job?
> Without any hints, it is very hard to reproduce and fix the bug.
>
> 2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <po...@okkam.it>:
>
>> Unfortunately not in public..moreover this error seems to be non
>> deterministic..sometime the programs runs correctly :(
>>
>> On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> Hmm, that issue must have been introduced as part of a recent change.
>>>
>>> Can you share your program so we can reproduce abs fix the bug?
>>> Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <po...@okkam.it>:
>>>
>>>> Hi to all,
>>>>
>>>> I'm running my flink-job on the new 0.9.0 version of Flink and now I
>>>> get this exception:
>>>>
>>>> Exception in thread "main"
>>>> org.apache.flink.optimizer.CompilerException: Bug: Plan generation for
>>>> Unions picked a ship strategy between binary plan operators.
>>>> at
>>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>>>> at
>>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>>>> at
>>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>>>> at
>>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>>>> at
>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>> at
>>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>>>> at
>>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>>>> at
>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>> at
>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>> at
>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>> at
>>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>>> at
>>>> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>>>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>>>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>>>> at
>>>> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
>>>> at
>>>> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
>>>> at
>>>> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
>>>>
>>>> How can I fix that?
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>
>>
>

Re: Datasets union CompilerException

Posted by Fabian Hueske <fh...@gmail.com>.
Hi, is it possible to get a smaller version of that program that reproduces
the bug or give a few more details about the structure of the job?
Without any hints, it is very hard to reproduce and fix the bug.

2015-06-24 18:23 GMT+02:00 Flavio Pompermaier <po...@okkam.it>:

> Unfortunately not in public..moreover this error seems to be non
> deterministic..sometime the programs runs correctly :(
>
> On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:
>
>> Hmm, that issue must have been introduced as part of a recent change.
>>
>> Can you share your program so we can reproduce abs fix the bug?
>> Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <po...@okkam.it>:
>>
>>> Hi to all,
>>>
>>> I'm running my flink-job on the new 0.9.0 version of Flink and now I get
>>> this exception:
>>>
>>> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
>>> Bug: Plan generation for Unions picked a ship strategy between binary plan
>>> operators.
>>> at
>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>>> at
>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>>> at
>>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>>> at
>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>>> at
>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>> at
>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>>> at
>>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>>> at
>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>> at
>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>> at
>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>> at
>>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>>> at
>>> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>>> at
>>> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
>>> at
>>> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
>>> at
>>> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
>>>
>>> How can I fix that?
>>>
>>> Best,
>>> Flavio
>>>
>>
>

Re: Datasets union CompilerException

Posted by Flavio Pompermaier <po...@okkam.it>.
Unfortunately not in public..moreover this error seems to be non
deterministic..sometime the programs runs correctly :(

On Wed, Jun 24, 2015 at 6:04 PM, Stephan Ewen <se...@apache.org> wrote:

> Hmm, that issue must have been introduced as part of a recent change.
>
> Can you share your program so we can reproduce abs fix the bug?
> Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <po...@okkam.it>:
>
>> Hi to all,
>>
>> I'm running my flink-job on the new 0.9.0 version of Flink and now I get
>> this exception:
>>
>> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
>> Bug: Plan generation for Unions picked a ship strategy between binary plan
>> operators.
>> at
>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
>> at
>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
>> at
>> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
>> at
>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
>> at
>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>> at
>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>> at
>> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
>> at
>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>> at
>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>> at
>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>> at
>> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
>> at
>> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
>> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
>> at
>> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
>> at
>> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
>> at
>> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
>>
>> How can I fix that?
>>
>> Best,
>> Flavio
>>
>

Re: Datasets union CompilerException

Posted by Stephan Ewen <se...@apache.org>.
Hmm, that issue must have been introduced as part of a recent change.

Can you share your program so we can reproduce abs fix the bug?
Am 24.06.2015 16:56 schrieb "Flavio Pompermaier" <po...@okkam.it>:

> Hi to all,
>
> I'm running my flink-job on the new 0.9.0 version of Flink and now I get
> this exception:
>
> Exception in thread "main" org.apache.flink.optimizer.CompilerException:
> Bug: Plan generation for Unions picked a ship strategy between binary plan
> operators.
> at
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.collect(BinaryUnionReplacer.java:113)
> at
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:72)
> at
> org.apache.flink.optimizer.traversals.BinaryUnionReplacer.postVisit(BinaryUnionReplacer.java:41)
> at
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:170)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
> at
> org.apache.flink.optimizer.plan.DualInputPlanNode.accept(DualInputPlanNode.java:163)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
> at
> org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:127)
> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:520)
> at org.apache.flink.optimizer.Optimizer.compile(Optimizer.java:402)
> at
> org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:173)
> at
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:54)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:789)
>
> How can I fix that?
>
> Best,
> Flavio
>