You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Flink Jira Bot (Jira)" <ji...@apache.org> on 2021/10/29 22:40:01 UTC
[jira] [Updated] (FLINK-20827) Just read record correlating to join
key in FilesystemLookUpFunc
[ https://issues.apache.org/jira/browse/FLINK-20827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Flink Jira Bot updated FLINK-20827:
-----------------------------------
Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized.
> Just read record correlating to join key in FilesystemLookUpFunc
> ----------------------------------------------------------------
>
> Key: FLINK-20827
> URL: https://issues.apache.org/jira/browse/FLINK-20827
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Connectors / Hive
> Reporter: zoucao
> Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> When using Temporal table join, all hive tables' records will be loaded into cache. But sometimes, the size of hive temporal table is larger than expected, and users can't know exactly how big it is in memory. In this situation, some error will occur, for example, `GC overhead limit exceeded`, `the heartbeat of TaskManager timeout (caused by gc)`.
> Maybe we can optimize the number of records readed from hive table? If the upstream records can be hashed only by using `Join key`, then we only need to load the part of records into cache, whose value of join key after being hashed, is equal to one fixed hash value. If it can be done, the whole table can be divided by the number of parallelism. I don't know whether it could come true in the upstream under the existing framework, but It is easy to support in `FileSystemLookupFunction`
> If not, we can add some logs to tell others the size of cache to help them to set MemorySize or other parameter of TM.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)