You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "jirawech.s (Jira)" <ji...@apache.org> on 2023/04/14 12:55:00 UTC
[jira] [Created] (FLINK-31811) Unsupported complex data type for Flink SQL
jirawech.s created FLINK-31811:
----------------------------------
Summary: Unsupported complex data type for Flink SQL
Key: FLINK-31811
URL: https://issues.apache.org/jira/browse/FLINK-31811
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.16.1
Reporter: jirawech.s
Fix For: 1.16.2
I found this issue when I tried to write data on local filesystem using Flink SQL
{code:java}
19:51:32,966 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact-operator (1/4) (4f2a09b638c786f74262c675d248afd9_80fe6c4f32f605d447b391cdb16cc1ff_0_4) switched from RUNNING to FAILED on 69ed2306-371b-4bfc-a98e-bf75fb41748f @ localhost (dataPort=-1).
java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:659) ~[?:1.8.0_301]
at java.util.ArrayList.get(ArrayList.java:435) ~[?:1.8.0_301]
at org.apache.parquet.schema.GroupType.getType(GroupType.java:216) ~[parquet-column-1.12.2.jar:1.12.2]
at org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector(ParquetSplitReaderUtil.java:523) ~[flink-parquet-1.16.1.jar:1.16.1]
at org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector(ParquetSplitReaderUtil.java:503) ~[flink-parquet-1.16.1.jar:1.16.1]
at org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createWritableVectors(ParquetVectorizedInputFormat.java:281) ~[flink-parquet-1.16.1.jar:1.16.1]
at org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createReaderBatch(ParquetVectorizedInputFormat.java:270) ~[flink-parquet-1.16.1.jar:1.16.1]
at org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createPoolOfBatches(ParquetVectorizedInputFormat.java:260) ~[flink-parquet-1.16.1.jar:1.16.1]
{code}
What i tried to do is writing complex data type to parquet file
Here is the schema of sink table. The problematic data type is ARRAY<MAP<STRING,STRING>>
{code:java}
CREATE TEMPORARY TABLE local_table (
`user_id` STRING, `order_id` STRING, `amount` INT, `restaurant_id` STRING, `experiment` ARRAY<MAP<STRING,STRING>>, `dt` STRING
) PARTITIONED BY (`dt`) WITH (
'connector'='filesystem',
'path'='file:///tmp/test_hadoop_write',
'format'='parquet',
'auto-compaction'='true',
'sink.partition-commit.policy.kind'='success-file'
) {code}
PS. It is used to work in Flink version 1.15.1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)