You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "luoyuxia (via GitHub)" <gi...@apache.org> on 2023/03/16 02:27:42 UTC
[GitHub] [flink] luoyuxia commented on a diff in pull request #21645: [FLINK-30556] Improve the logic for enumerating splits for Hive source to avoid potential OOM
luoyuxia commented on code in PR #21645:
URL: https://github.com/apache/flink/pull/21645#discussion_r1138014049
##########
flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/src/assigners/LocalityAwareSplitAssigner.java:
##########
@@ -69,9 +70,10 @@ public class LocalityAwareSplitAssigner implements FileSplitAssigner {
// --------------------------------------------------------------------------------------------
- public LocalityAwareSplitAssigner(Collection<FileSourceSplit> splits) {
- for (FileSourceSplit split : splits) {
- this.unassigned.add(new SplitWithInfo(split));
+ public LocalityAwareSplitAssigner(Iterator<FileSourceSplit> splits) {
+ while (splits.hasNext()) {
Review Comment:
Actaully, it'll still cause OOM for in here we will still maintain **all splits** in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org