You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Jingsong Lee (Jira)" <ji...@apache.org> on 2021/06/17 04:25:00 UTC

[jira] [Comment Edited] (FLINK-22994) Improve the performance of invoking nesting udf

    [ https://issues.apache.org/jira/browse/FLINK-22994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364676#comment-17364676 ] 

Jingsong Lee edited comment on FLINK-22994 at 6/17/21, 4:24 AM:
----------------------------------------------------------------

[~zicat] I didn't say the conversion is no cost, its cost is very low.

Your testing [^StringConverterTest.java] is inaccurate. I just modify it like this:
{code:java}
public static void main(String[] args) {
    final int totalTestCount = 100 * 10000;
    System.out.println(BinaryStringData.fromString(""));
    System.out.println(
            "Total test count:"
                    + totalTestCount
                    + ", The spend with MapMapConverter "
                    + testWithMapMapConverter(totalTestCount)
                    + " ms.");
    System.out.println(
            "Total test count:"
                    + totalTestCount
                    + ", The spend without Converter "
                    + testWithOutMapMapConverter(totalTestCount)
                    + " ms.");
}
{code}
The result is different.

Load BinaryStringData and its related classes to the classloader. This is very important.


was (Author: lzljs3620320):
[~zicat] I didn't say the conversion is no cost, its cost is very low. Your testing [^StringConverterTest.java] is meaningless, its other cost is so low.

> Improve the performance of invoking nesting udf
> -----------------------------------------------
>
>                 Key: FLINK-22994
>                 URL: https://issues.apache.org/jira/browse/FLINK-22994
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / Runtime
>    Affects Versions: 1.12.4
>         Environment: h5.  
>            Reporter: lynn1.zhang
>            Assignee: lynn1.zhang
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: StringConverterTest.java, Test.java, image-2021-06-15-15-18-12-619.png, image-2021-06-15-15-19-01-103.png, image-2021-06-15-15-27-26-739.png, image-2021-06-15-15-28-28-137.png, image-2021-06-15-15-29-09-773.png, image-2021-06-15-15-30-14-775.png, image-2021-06-15-15-42-08-065.png, new_projection_code, old_projection_code, test.sql
>
>
> h1. BackGround
> Flink maintain the udf result as BinaryData, like BinaryStringData. When invoking nesting udf like select useless(int_ip_2_string(ip)), the result of int_ip_2_string(ip) will be toInternalOrNull and toExternal.
> Below is the Generated Code
> !image-2021-06-15-15-18-12-619.png!   This issue will improve it as below
> !image-2021-06-15-15-19-01-103.png!
> h1. Performance Compare
> Condition: Source = Kafka, Schema = PB with snappy; Flink Slot = 1; taskmanager.memory.process.size=4g; Linux Core = Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
>  UDF Introduction:
>  * ipip:  input: int ip, output: map ip_info, map size = 14.
>  * ip_2_country: input map ip_info, output: string country.
>  * ip_2_region: input  map ip_info, output: string region.
>  * ip_2_isp_domain: input  map ip_info, output: string isp.
>  * ip_2_timezone: input map ip_info, output: string timezone.
> h5. The throughput without udf invoke: 764.50 k/s
> !image-2021-06-15-15-27-26-739.png!
> h5. The throughput with udf invoke: 183.24 k/s
> !image-2021-06-15-15-42-08-065.png!
> h5. The throughput with udf nesting invoke: 41.42 k/s
> !image-2021-06-15-15-29-09-773.png!
> h5. The throughput with udf nesting invoke after this issue: 174.41 k/s
> !image-2021-06-15-15-30-14-775.png!
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)