You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by tanjialiang <ta...@126.com> on 2023/04/13 02:05:39 UTC
[DISCUSS] EncodingFormat and DecondingFormat provide copy API
Hi, devs.
I'd like to start a discussion about to EncodingFormat and DecondingFormat provide copy API, which relate to FLINK-31686 [1].
Current, DecodingFormat doesn't support copy(), which makes the DecodingFormat resued after filter/projection is pushed down. The EncodingFormat has the same problem if class implements EncodingFormat#applyWritableMetadata(). So I think EncodingFormat and DecodingFormat need to provide a copy function, and it should be a deep copy if format implements DecodingFormat#applyReadableMetadata/EncodingFormat#applyWritableMetadata/BulkDecodingFormat#applyFilters.
Looking forwards to your feedback.
[1]: [https://issues.apache.org/jira/browse/FLINK-31686]
Best regards,
tanjialiang
Re: [DISCUSS] EncodingFormat and DecondingFormat provide copy API
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Tanjialiang,
Like we discussed in another thread, please feel free to create a FLIP and
start further discussion. In case you need any access right to the wiki
page, please let me know, thanks.
Best regards,
Jing
On Sun, Apr 23, 2023 at 4:23 AM tanjialiang <ta...@126.com> wrote:
> Hi, devs.
> Do anyone has any question about this discussion? I'm looking forwards to
> your feedback.
>
>
>
> Best regards,
> tanjialiang.
>
>
> ---- Replied Message ----
> | From | tanjialiang<ta...@126.com> |
> | Date | 4/13/2023 10:05 |
> | To | dev@flink.apache.org<de...@flink.apache.org> |
> | Subject | [DISCUSS] EncodingFormat and DecondingFormat provide copy API |
> Hi, devs.
>
>
> I'd like to start a discussion about to EncodingFormat and DecondingFormat
> provide copy API, which relate to FLINK-31686 [1].
>
>
> Current, DecodingFormat doesn't support copy(), which makes the
> DecodingFormat resued after filter/projection is pushed down. The
> EncodingFormat has the same problem if class implements
> EncodingFormat#applyWritableMetadata(). So I think EncodingFormat and
> DecodingFormat need to provide a copy function, and it should be a deep
> copy if format implements
> DecodingFormat#applyReadableMetadata/EncodingFormat#applyWritableMetadata/BulkDecodingFormat#applyFilters.
>
>
>
> Looking forwards to your feedback.
>
>
> [1]: [https://issues.apache.org/jira/browse/FLINK-31686]
>
>
> Best regards,
> tanjialiang
>
>
>
>
Re: [DISCUSS] EncodingFormat and DecondingFormat provide copy API
Posted by tanjialiang <ta...@126.com>.
Hi, devs.
Do anyone has any question about this discussion? I'm looking forwards to your feedback.
Best regards,
tanjialiang.
---- Replied Message ----
| From | tanjialiang<ta...@126.com> |
| Date | 4/13/2023 10:05 |
| To | dev@flink.apache.org<de...@flink.apache.org> |
| Subject | [DISCUSS] EncodingFormat and DecondingFormat provide copy API |
Hi, devs.
I'd like to start a discussion about to EncodingFormat and DecondingFormat provide copy API, which relate to FLINK-31686 [1].
Current, DecodingFormat doesn't support copy(), which makes the DecodingFormat resued after filter/projection is pushed down. The EncodingFormat has the same problem if class implements EncodingFormat#applyWritableMetadata(). So I think EncodingFormat and DecodingFormat need to provide a copy function, and it should be a deep copy if format implements DecodingFormat#applyReadableMetadata/EncodingFormat#applyWritableMetadata/BulkDecodingFormat#applyFilters.
Looking forwards to your feedback.
[1]: [https://issues.apache.org/jira/browse/FLINK-31686]
Best regards,
tanjialiang
Re: [DISCUSS] EncodingFormat and DecondingFormat provide copy API
Posted by tanjialiang <ta...@126.com>.
Hi, Aitozi,
Thanks you for your feedback, and +1 for your point of view.
Best,
tanjialiang.
---- Replied Message ----
| From | Aitozi<gj...@gmail.com> |
| Date | 4/14/2023 16:19 |
| To | <de...@flink.apache.org> |
| Subject | Re: [DISCUSS] EncodingFormat and DecondingFormat provide copy API |
Hi tanjialiang
Thanks for reporting this, I think it's a reasonable requirements.
The problem might be introduced during the optimization when
reusing the mutable state in Source. So the DecodingFormat#copy can
avoid this situation.
But after checking the related code for DynamicTableSink, It's a
little different, I think it do not rely on the copy interface now,
even the DynamicTableSink#copy is not invoked by the framework. I
guess the reason is that the optimization for sink are all static
abilities
spec which are all done before the optimization. So no copy is happened.
Anyway, it's unclear for the implementer to know which stage will the
`apply` happen, So +1 for introducing the copy for EncodingFormat and
DecodingFormat to align the supporting of deep copy semantic this time.
Best,
Aitozi.
tanjialiang <ta...@126.com> 于2023年4月13日周四 10:05写道：
Hi, devs.
I'd like to start a discussion about to EncodingFormat and DecondingFormat provide copy API, which relate to FLINK-31686 [1].
Current, DecodingFormat doesn't support copy(), which makes the DecodingFormat resued after filter/projection is pushed down. The EncodingFormat has the same problem if class implements EncodingFormat#applyWritableMetadata(). So I think EncodingFormat and DecodingFormat need to provide a copy function, and it should be a deep copy if format implements DecodingFormat#applyReadableMetadata/EncodingFormat#applyWritableMetadata/BulkDecodingFormat#applyFilters.
Looking forwards to your feedback.
[1]: [https://issues.apache.org/jira/browse/FLINK-31686]
Best regards,
tanjialiang
Re: [DISCUSS] EncodingFormat and DecondingFormat provide copy API
Posted by Aitozi <gj...@gmail.com>.
Hi tanjialiang
Thanks for reporting this, I think it's a reasonable requirements.
The problem might be introduced during the optimization when
reusing the mutable state in Source. So the DecodingFormat#copy can
avoid this situation.
But after checking the related code for DynamicTableSink, It's a
little different, I think it do not rely on the copy interface now,
even the DynamicTableSink#copy is not invoked by the framework. I
guess the reason is that the optimization for sink are all static
abilities
spec which are all done before the optimization. So no copy is happened.
Anyway, it's unclear for the implementer to know which stage will the
`apply` happen, So +1 for introducing the copy for EncodingFormat and
DecodingFormat to align the supporting of deep copy semantic this time.
Best,
Aitozi.
tanjialiang <ta...@126.com> 于2023年4月13日周四 10:05写道：
>
> Hi, devs.
>
>
> I'd like to start a discussion about to EncodingFormat and DecondingFormat provide copy API, which relate to FLINK-31686 [1].
>
>
> Current, DecodingFormat doesn't support copy(), which makes the DecodingFormat resued after filter/projection is pushed down. The EncodingFormat has the same problem if class implements EncodingFormat#applyWritableMetadata(). So I think EncodingFormat and DecodingFormat need to provide a copy function, and it should be a deep copy if format implements DecodingFormat#applyReadableMetadata/EncodingFormat#applyWritableMetadata/BulkDecodingFormat#applyFilters.
>
>
>
> Looking forwards to your feedback.
>
>
> [1]: [https://issues.apache.org/jira/browse/FLINK-31686]
>
>
> Best regards,
> tanjialiang
>
>
>