You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by shkob1 <sh...@gmail.com> on 2019/05/14 18:23:57 UTC
Table program cannot be compiled
Hey,
While running a SQL query i get an OutOfMemoryError exception and "Table
program cannot be compiled" [2].
In my scenario i'm trying to enrich an event using an array of tags, each
tag has a boolean classification (like a WHERE clause) and with a custom
function i'm filtering the array to keep only TRUE results.
While i cannot publish the actual query the form of the query is as follows:
/SELECT originalEvent, conditionalArray('Tag1', boolean_condition1, 'Tag2',
boolean_condition2).. FROM../
(more info at the use case here[1] )
I do have quite a lot of those tags (counting 126 tags now) but i had
something similar in the past without this error.
Any idea about it and if you can think of a workaround to this issue?
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Reconstruct-object-through-partial-select-query-td27782.html
[2]
*org.apache.flink.api.common.InvalidProgramException: Table program cannot
be compiled. This is a bug. Please file an issue.*
at org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36)
at
org.apache.flink.table.runtime.CRowProcessRunner.compile(CRowProcessRunner.scala:35)
at
org.apache.flink.table.runtime.CRowProcessRunner.open(CRowProcessRunner.scala:49)
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
at
org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
at java.lang.Thread.run(Thread.java:745)
*Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded*
at java.util.HashMap.newNode(HashMap.java:1734)
at java.util.HashMap.putVal(HashMap.java:641)
at java.util.HashMap.putMapEntries(HashMap.java:514)
at java.util.HashMap.putAll(HashMap.java:784)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3322)
at org.codehaus.janino.UnitCompiler.access$4900(UnitCompiler.java:212)
at
org.codehaus.janino.UnitCompiler$8.visitLocalVariableDeclarationStatement(UnitCompiler.java:3207)
at
org.codehaus.janino.UnitCompiler$8.visitLocalVariableDeclarationStatement(UnitCompiler.java:3175)
at
org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3348)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3174)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3231)
at org.codehaus.janino.UnitCompiler.access$3800(UnitCompiler.java:212)
at org.codehaus.janino.UnitCompiler$8.visitBlock(UnitCompiler.java:3193)
at org.codehaus.janino.UnitCompiler$8.visitBlock(UnitCompiler.java:3175)
at org.codehaus.janino.Java$Block.accept(Java.java:2753)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3174)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3267)
at org.codehaus.janino.UnitCompiler.access$4200(UnitCompiler.java:212)
at
org.codehaus.janino.UnitCompiler$8.visitIfStatement(UnitCompiler.java:3197)
at
org.codehaus.janino.UnitCompiler$8.visitIfStatement(UnitCompiler.java:3175)
at org.codehaus.janino.Java$IfStatement.accept(Java.java:2923)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3174)
at
org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:3163)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2986)
at
org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1313)
at
org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1286)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:785)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:436)
at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:212)
at
org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:390)
at
org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:385)
at
org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1405)
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Table program cannot be compiled
Posted by Timo Walther <tw...@apache.org>.
Hi Shahar,
yes the number of parameters should be the issue for a cannot compile
exception. If you moved most of the constants to a member in the
function, it should actually work.
Do you have a little reproducible example somewhere?
Thanks,
Timo
Am 16.05.19 um 19:59 schrieb shkob1:
> Hi Timo,
>
> Thanks for the link.
> Not sure i understand your suggestion though, is the goal here reducing the
> amount of parameters coming to the UDF? if thats the case i can maybe have
> the tag names there, but still need the expressions to get evaluated before
> entering the eval. Do you see this in a different way?
>
> I tried moving the tag names to be a member within the function instead of a
> parameter, but apparently i still have too many arguments.
>
> Let me know if this is not what you meant.
>
> Thanks!
> Shahar
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Table program cannot be compiled
Posted by shkob1 <sh...@gmail.com>.
Hi Timo,
Thanks for the link.
Not sure i understand your suggestion though, is the goal here reducing the
amount of parameters coming to the UDF? if thats the case i can maybe have
the tag names there, but still need the expressions to get evaluated before
entering the eval. Do you see this in a different way?
I tried moving the tag names to be a member within the function instead of a
parameter, but apparently i still have too many arguments.
Let me know if this is not what you meant.
Thanks!
Shahar
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Table program cannot be compiled
Posted by Timo Walther <tw...@apache.org>.
Hi,
too many arguments for calling a UDF could currently lead to "grows
beyond 64 KB" and maybe also causes the GC exception. This is a known
issue covered in https://issues.apache.org/jira/browse/FLINK-8921.
Could you also add the tags to the function itself? Maybe as a static
map for constant time access outside of the eval method?
Regards,
Timo
Am 15.05.19 um 17:10 schrieb Andrey Zagrebin:
> Hi, I am looping in Timo and Dawid to look at the problem.
>
> On Tue, May 14, 2019 at 9:12 PM shkob1 <shahar.kobrinsky@gmail.com
> <ma...@gmail.com>> wrote:
>
> BTW looking at past posts on this issue[1] it should have been
> fixed? i'm
> using version 1.7.2
> Also the recommendation was to use a custom function, though
> that's exactly
> what im doing with the conditionalArray function[2]
>
> Thanks!
>
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataStreamCalcRule-1802-quot-grows-beyond-64-KB-when-execute-long-sql-td20832.html#a20841
>
> [2]
> public class ConditionalArrayFunction extends ScalarFunction {
>
> public static final String NAME = "conditionalArray";
>
> public String[] eval(Object... keyValues) {
> if (keyValues.length == 0) {
> return new String[]{};
> }
> final List keyValuesList = Arrays.asList(keyValues);
> List<String> trueItems = Lists.newArrayList();
> for (int i = 0; i < keyValuesList.size(); i = i + 2){
> final String key = (String)keyValuesList.get(i);
> final Object value = keyValuesList.get(i + 1);
>
> if (value != null && (boolean)value)
> trueItems.add(key);
> }
> return trueItems.toArray(new String[0]);
> }
> }
>
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Table program cannot be compiled
Posted by Andrey Zagrebin <an...@ververica.com>.
Hi, I am looping in Timo and Dawid to look at the problem.
On Tue, May 14, 2019 at 9:12 PM shkob1 <sh...@gmail.com> wrote:
> BTW looking at past posts on this issue[1] it should have been fixed? i'm
> using version 1.7.2
> Also the recommendation was to use a custom function, though that's exactly
> what im doing with the conditionalArray function[2]
>
> Thanks!
>
> [1]
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataStreamCalcRule-1802-quot-grows-beyond-64-KB-when-execute-long-sql-td20832.html#a20841
>
> [2]
> public class ConditionalArrayFunction extends ScalarFunction {
>
> public static final String NAME = "conditionalArray";
>
> public String[] eval(Object... keyValues) {
> if (keyValues.length == 0) {
> return new String[]{};
> }
> final List keyValuesList = Arrays.asList(keyValues);
> List<String> trueItems = Lists.newArrayList();
> for (int i = 0; i < keyValuesList.size(); i = i + 2){
> final String key = (String)keyValuesList.get(i);
> final Object value = keyValuesList.get(i + 1);
>
> if (value != null && (boolean)value)
> trueItems.add(key);
> }
> return trueItems.toArray(new String[0]);
> }
> }
>
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Table program cannot be compiled
Posted by shkob1 <sh...@gmail.com>.
BTW looking at past posts on this issue[1] it should have been fixed? i'm
using version 1.7.2
Also the recommendation was to use a custom function, though that's exactly
what im doing with the conditionalArray function[2]
Thanks!
[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DataStreamCalcRule-1802-quot-grows-beyond-64-KB-when-execute-long-sql-td20832.html#a20841
[2]
public class ConditionalArrayFunction extends ScalarFunction {
public static final String NAME = "conditionalArray";
public String[] eval(Object... keyValues) {
if (keyValues.length == 0) {
return new String[]{};
}
final List keyValuesList = Arrays.asList(keyValues);
List<String> trueItems = Lists.newArrayList();
for (int i = 0; i < keyValuesList.size(); i = i + 2){
final String key = (String)keyValuesList.get(i);
final Object value = keyValuesList.get(i + 1);
if (value != null && (boolean)value)
trueItems.add(key);
}
return trueItems.toArray(new String[0]);
}
}
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Table program cannot be compiled
Posted by shkob1 <sh...@gmail.com>.
In a subsequent run i get
Caused by: org.codehaus.janino.JaninoRuntimeException: Code of method
"split$3681$(LDataStreamCalcRule$3682;)V" of class "DataStreamCalcRule$3682"
grows beyond 64 KB
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/