You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jingsong Lee (Jira)" <ji...@apache.org> on 2021/07/09 02:23:00 UTC
[jira] [Closed] (FLINK-22994) Improve the performance of nesting
udf invoking
[ https://issues.apache.org/jira/browse/FLINK-22994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jingsong Lee closed FLINK-22994.
--------------------------------
Fix Version/s: 1.14.0
Resolution: Fixed
Implements via:
master: 31fcb6c22baaac1308a72559e32e2d089fc8fcad
> Improve the performance of nesting udf invoking
> -----------------------------------------------
>
> Key: FLINK-22994
> URL: https://issues.apache.org/jira/browse/FLINK-22994
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Affects Versions: 1.12.4
> Environment: h5.
> Reporter: lynn1.zhang
> Assignee: lynn1.zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: StringConverterTest.java, Test.java, image-2021-06-15-15-18-12-619.png, image-2021-06-15-15-19-01-103.png, image-2021-06-15-15-27-26-739.png, image-2021-06-15-15-28-28-137.png, image-2021-06-15-15-29-09-773.png, image-2021-06-15-15-30-14-775.png, image-2021-06-15-15-42-08-065.png, new_projection_code, old_projection_code, test.sql
>
>
> h1. BackGround
> In some nesting udf invoking cases, Flink convert the udf result to external object and then convert to internalOrNull object as params for next udf invoking. The performance of some converter is poor.
> h1. Test Params
> Source = Kafka, Schema = PB with snappy; Flink Slot = 1; taskmanager.memory.process.size=4g; Linux Core = Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
> UDF Introduction:
> * ipip: input: int ip, output: map ip_info, map size = 14.
> * ip_2_country: input map ip_info, output: string country.
> * ip_2_region: input map ip_info, output: string region.
> * ip_2_isp_domain: input map ip_info, output: string isp.
> * ip_2_timezone: input map ip_info, output: string timezone.
> h1. Performance Compare with MapMapConverter
> h5. The throughput of nesting udf invoking with MapMapConverter(5 times): 41.42 k/s
> !image-2021-06-15-15-29-09-773.png!
> h1. Goal
> For some cases, skip toInternalOrNull & toExternal, Use the udf result directly.
> h1. Performance Compare without MapMapConverter
> h5. The throughput of nesting udf invoking without MapMapConverter: 174.41 k/s
> !image-2021-06-15-15-30-14-775.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)