You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by allanqinjy <al...@163.com> on 2020/05/18 08:36:22 UTC

数组越界

各位好，
     flink1.10,在跑flink批量sql的适合语法通过没问题，在运行脚步的适合报错如下，hive脚步跑没有问题，不知道为什么flink 跑会报数组越界，这个是什么问题？

Re: 数组越界

Posted by Leonard Xu <xb...@gmail.com>.
Hi,  allanqinjy

方便贴下查询的query吗？今天在排查另外一个问题时也遇到了这个问题，我建了issue来跟踪[1]，想看下是不是相同原因。


Best,
Leonard
[1] https://issues.apache.org/jira/browse/FLINK-17847 <https://issues.apache.org/jira/browse/FLINK-17847>

> 在 2020年5月18日，19:52，Leonard Xu <xb...@gmail.com> 写道：
> 
> Hi, allanqinjy
> 
> 运行时抛ArrayIndexOutOfBoundsException 是不符合预期的，感觉是个bug。
> 可以复现的haul，方便提供下复现的sql和数据吗？
> 
> Best,
> Leonard Xu
> 
> 
>> 在 2020年5月18日，17:37，Benchao Li <li...@gmail.com> 写道：
>> 
>> 数组长度是运行时的问题，编译期并不知道数组的长度。而且现在好像也没有检查下标是不是合法（比如必须大于0）。我们以前也经常遇到这种问题。
>> 
>> allanqinjy <al...@163.com> 于2020年5月18日周一 下午5:15写道：
>> 
>>> 我觉得要是从1开始，那么编译的时候就应该报异常了，而不是提交作业运行报。
>>> 
>>> 
>>> 
>>> 
>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 22369621
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.table.runtime.util.SegmentsUtil.getByteMultiSegments(SegmentsUtil.java:598)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.table.runtime.util.SegmentsUtil.getByte(SegmentsUtil.java:590)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.table.runtime.util.SegmentsUtil.bitGet(SegmentsUtil.java:534)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.table.dataformat.BinaryArray.isNullAt(BinaryArray.java:117)
>>> 18-05-2020 16:27:14 CST INFO -  at BatchCalc$822.processElement(Unknown
>>> Source)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:550)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:527)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:487)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:748)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:734)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:93)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
>>> 18-05-2020 16:27:14 CST INFO -  at
>>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 在 2020-05-18 16:38:16，"1048262223" <10...@qq.com> 写道：
>>>> 图看不到
>>>> flink内置udf和hive udf不同，有些udf下标是从1开始的
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 各位好，
>>>> &nbsp; &nbsp;
>>> &nbsp;flink1.10,在跑flink批量sql的适合语法通过没问题，在运行脚步的适合报错如下，hive脚步跑没有问题，不知道为什么flink
>>> 跑会报数组越界，这个是什么问题？
>>>> &nbsp; &nbsp; &nbsp;
>>>> 
>>>> 
>>>> 
>>>> 
>>>> &nbsp;
>>> 
>> 
>> 
>> -- 
>> 
>> Benchao Li
>> School of Electronics Engineering and Computer Science, Peking University
>> Tel:+86-15650713730
>> Email: libenchao@gmail.com; libenchao@pku.edu.cn
>

Re: 数组越界

Posted by Leonard Xu <xb...@gmail.com>.
Hi, allanqinjy

运行时抛ArrayIndexOutOfBoundsException 是不符合预期的，感觉是个bug。
可以复现的haul，方便提供下复现的sql和数据吗？

Best,
Leonard Xu


> 在 2020年5月18日，17:37，Benchao Li <li...@gmail.com> 写道：
> 
> 数组长度是运行时的问题，编译期并不知道数组的长度。而且现在好像也没有检查下标是不是合法（比如必须大于0）。我们以前也经常遇到这种问题。
> 
> allanqinjy <al...@163.com> 于2020年5月18日周一 下午5:15写道：
> 
>> 我觉得要是从1开始，那么编译的时候就应该报异常了，而不是提交作业运行报。
>> 
>> 
>> 
>> 
>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 22369621
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.table.runtime.util.SegmentsUtil.getByteMultiSegments(SegmentsUtil.java:598)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.table.runtime.util.SegmentsUtil.getByte(SegmentsUtil.java:590)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.table.runtime.util.SegmentsUtil.bitGet(SegmentsUtil.java:534)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.table.dataformat.BinaryArray.isNullAt(BinaryArray.java:117)
>> 18-05-2020 16:27:14 CST INFO -  at BatchCalc$822.processElement(Unknown
>> Source)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:550)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:527)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:487)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:748)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:734)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:93)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
>> 18-05-2020 16:27:14 CST INFO -  at
>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 在 2020-05-18 16:38:16，"1048262223" <10...@qq.com> 写道：
>>> 图看不到
>>> flink内置udf和hive udf不同，有些udf下标是从1开始的
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 各位好，
>>> &nbsp; &nbsp;
>> &nbsp;flink1.10,在跑flink批量sql的适合语法通过没问题，在运行脚步的适合报错如下，hive脚步跑没有问题，不知道为什么flink
>> 跑会报数组越界，这个是什么问题？
>>> &nbsp; &nbsp; &nbsp;
>>> 
>>> 
>>> 
>>> 
>>> &nbsp;
>> 
> 
> 
> -- 
> 
> Benchao Li
> School of Electronics Engineering and Computer Science, Peking University
> Tel:+86-15650713730
> Email: libenchao@gmail.com; libenchao@pku.edu.cn

Re: 回复：数组越界

Posted by Benchao Li <li...@gmail.com>.
数组长度是运行时的问题，编译期并不知道数组的长度。而且现在好像也没有检查下标是不是合法（比如必须大于0）。我们以前也经常遇到这种问题。

allanqinjy <al...@163.com> 于2020年5月18日周一 下午5:15写道：

> 我觉得要是从1开始，那么编译的时候就应该报异常了，而不是提交作业运行报。
>
>
>
>
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 22369621
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.table.runtime.util.SegmentsUtil.getByteMultiSegments(SegmentsUtil.java:598)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.table.runtime.util.SegmentsUtil.getByte(SegmentsUtil.java:590)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.table.runtime.util.SegmentsUtil.bitGet(SegmentsUtil.java:534)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.table.dataformat.BinaryArray.isNullAt(BinaryArray.java:117)
> 18-05-2020 16:27:14 CST INFO -  at BatchCalc$822.processElement(Unknown
> Source)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:550)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:527)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:487)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:748)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:734)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:93)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
> 18-05-2020 16:27:14 CST INFO -  at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
>
>
>
>
>
>
>
>
> 在 2020-05-18 16:38:16，"1048262223" <10...@qq.com> 写道：
> >图看不到
> >flink内置udf和hive udf不同，有些udf下标是从1开始的
> >
> >
> >
> >
> >
> >
> >
> >
> >各位好，
> >&nbsp; &nbsp;
> &nbsp;flink1.10,在跑flink批量sql的适合语法通过没问题，在运行脚步的适合报错如下，hive脚步跑没有问题，不知道为什么flink
> 跑会报数组越界，这个是什么问题？
> >&nbsp; &nbsp; &nbsp;
> >
> >
> >
> >
> >&nbsp;
>


-- 

Benchao Li
School of Electronics Engineering and Computer Science, Peking University
Tel:+86-15650713730
Email: libenchao@gmail.com; libenchao@pku.edu.cn

Re:回复：数组越界

Posted by allanqinjy <al...@163.com>.
我觉得要是从1开始，那么编译的时候就应该报异常了，而不是提交作业运行报。




Caused by: java.lang.ArrayIndexOutOfBoundsException: 22369621
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.table.runtime.util.SegmentsUtil.getByteMultiSegments(SegmentsUtil.java:598)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.table.runtime.util.SegmentsUtil.getByte(SegmentsUtil.java:590)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.table.runtime.util.SegmentsUtil.bitGet(SegmentsUtil.java:534)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.table.dataformat.BinaryArray.isNullAt(BinaryArray.java:117)
18-05-2020 16:27:14 CST INFO - 	at BatchCalc$822.processElement(Unknown Source)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.pushToOperator(OperatorChain.java:550)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:527)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.collect(OperatorChain.java:487)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:748)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingBroadcastingOutputCollector.collect(OperatorChain.java:734)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
18-05-2020 16:27:14 CST INFO -	at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:93)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
18-05-2020 16:27:14 CST INFO - 	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)








在 2020-05-18 16:38:16，"1048262223" <10...@qq.com> 写道：
>图看不到
>flink内置udf和hive udf不同，有些udf下标是从1开始的
>
>
>
>
>
>
>
>
>各位好，
>&nbsp; &nbsp; &nbsp;flink1.10,在跑flink批量sql的适合语法通过没问题，在运行脚步的适合报错如下，hive脚步跑没有问题，不知道为什么flink 跑会报数组越界，这个是什么问题？
>&nbsp; &nbsp; &nbsp;
>
>
>
>
>&nbsp;

回复：数组越界

Posted by 1048262223 <10...@qq.com>.
图看不到
flink内置udf和hive udf不同，有些udf下标是从1开始的








各位好，
&nbsp; &nbsp; &nbsp;flink1.10,在跑flink批量sql的适合语法通过没问题，在运行脚步的适合报错如下，hive脚步跑没有问题，不知道为什么flink 跑会报数组越界，这个是什么问题？
&nbsp; &nbsp; &nbsp;




&nbsp;