You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Lian Jiang (Jira)" <ji...@apache.org> on 2020/10/12 15:39:00 UTC
[jira] [Comment Edited] (FLINK-19491) AvroSerializerSnapshot cannot
handle large schema
[ https://issues.apache.org/jira/browse/FLINK-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211732#comment-17211732 ]
Lian Jiang edited comment on FLINK-19491 at 10/12/20, 3:38 PM:
---------------------------------------------------------------
Thanks [~nicholasjiang] and [~AHeise] for taking a look. Could this issue be fixed for 1.12? Appreciate sharing the plan.
was (Author: jiangok):
Thanks [~nicholasjiang] and [~AHeise] for taking a look. Could this issue been fixed for 1.12? Appreciate sharing the plan.
> AvroSerializerSnapshot cannot handle large schema
> -------------------------------------------------
>
> Key: FLINK-19491
> URL: https://issues.apache.org/jira/browse/FLINK-19491
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.10.2, 1.12.0, 1.11.2
> Reporter: Arvid Heise
> Priority: Major
>
> Flink can only handle schemas up to a size of 64kb.
>
> {noformat}
> Caused by: java.io.UTFDataFormatException: encoded string too long: 223502 bytes
> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:364)
> at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
> at org.apache.flink.formats.avro.typeutils.AvroSerializerSnapshot.writeSnapshot(AvroSerializerSnapshot.java:75)
> at org.apache.flink.api.common.typeutils.TypeSerializerSnapshot.writeVersionedSnapshot(TypeSerializerSnapshot.java:153)
> at org.apache.flink.api.common.typeutils.NestedSerializersSnapshotDelegate.writeNestedSerializerSnapshots(NestedSerializersSnapshotDelegate.java:159)
> at org.apache.flink.api.common.typeutils.CompositeTypeSerializerSnapshot.writeSnapshot(CompositeTypeSerializerSnapshot.java:148)
> at org.apache.flink.api.common.typeutils.TypeSerializerSnapshot.writeVersionedSnapshot(TypeSerializerSnapshot.java:153)
> at org.apache.flink.api.common.typeutils.TypeSerializerSnapshotSerializationUtil$TypeSerializerSnapshotSerializationProxy.write(TypeSerializerSnapshotSerializationUtil.java:138)
> at org.apache.flink.api.common.typeutils.TypeSerializerSnapshotSerializationUtil.writeSerializerSnapshot(TypeSerializerSnapshotSerializationUtil.java:55)
> at org.apache.flink.runtime.state.metainfo.StateMetaInfoSnapshotReadersWriters$CurrentWriterImpl.writeStateMetaInfoSnapshot(StateMetaInfoSnapshotReadersWriters.java:183)
> at org.apache.flink.runtime.state.KeyedBackendSerializationProxy.write(KeyedBackendSerializationProxy.java:126)
> at org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:171)
> at org.apache.flink.runtime.state.heap.HeapSnapshotStrategy$1.callInternal(HeapSnapshotStrategy.java:158)
> at org.apache.flink.runtime.state.AsyncSnapshotCallable.call(AsyncSnapshotCallable.java:75)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:510)
> ... 5 common frames omitted{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)