You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "tanjialiang (Jira)" <ji...@apache.org> on 2023/04/01 08:27:00 UTC
[jira] [Updated] (FLINK-31686) Connector should replace the shallow copy with deep copy
[ https://issues.apache.org/jira/browse/FLINK-31686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tanjialiang updated FLINK-31686:
--------------------------------
Summary: Connector should replace the shallow copy with deep copy (was: All of connector should replace the shallow copy with deep copy)
> Connector should replace the shallow copy with deep copy
> --------------------------------------------------------
>
> Key: FLINK-31686
> URL: https://issues.apache.org/jira/browse/FLINK-31686
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.16.1
> Reporter: tanjialiang
> Priority: Major
> Attachments: image-2023-04-01-16-18-48-762.png, image-2023-04-01-16-18-56-075.png
>
>
> Hi team, when i using the following sql
> {code:java}
> CREATE TABLE student (
> `id` STRING,
> `name` STRING,
> `age` INT
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = '...',
> 'format' = 'orc'
> );
> select
> t1.total,
> t2.total
> from
> (
> select
> count(*) as total,
> 1 as join_key
> from student
> where name = 'tanjialiang'
> ) t1
> LEFT JOIN (
> select
> count(*) as total,
> 1 as join_key
> from student;
> ) t2
> ON t1.join_key = t2.join_key; {code}
>
> it will throw an error
> !image-2023-04-01-16-18-48-762.png!
>
> I tried to solve it, and i found filesystem connector's copy function using a shallow copy instread of deep copy. It lead to all of query from a same table source reuse the same bulkWriterFormat, and my query have filter condition, which will push down into the bulkWriterFormat, so the filter condition maybe reuse.
> I found the DynamicTableSource and DynamicTableSink's copy function comment to ask we should impletement it with deep copy, but i found every connector are using shallow copy to impletement it. So i think not only the filesystem connector have this problem.
> !image-2023-04-01-16-18-56-075.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)