You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "HideOnBush (Jira)" <ji...@apache.org> on 2021/01/26 07:15:00 UTC

[jira] [Created] (FLINK-21145) Flink Temporal Join Hive optimization

HideOnBush created FLINK-21145:
----------------------------------

             Summary: Flink Temporal Join Hive optimization
                 Key: FLINK-21145
                 URL: https://issues.apache.org/jira/browse/FLINK-21145
             Project: Flink
          Issue Type: Wish
          Components: Connectors / Hive
    Affects Versions: 1.12.0
            Reporter: HideOnBush


When flink temporal join hive dimension table, the latest partition data will be loaded into task memory in full, which will lead to high memory overhead. In fact, sometimes the latest full data is not required. You can add options like options in future versions. Is the dimension table data filtered?
For example, select * from dim /*'streaming-source.partition.include' ='latest' condition='fild1=ab'*/ filter the latest partition data as long as fild1=ab



--
This message was sent by Atlassian Jira
(v8.3.4#803005)