[jira] [Updated] (FLINK-29617) Cost too much time to start SourceCoordinator of hdfsFileSource when start JobMaster

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-29617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-29617:
-----------------------------------
    Labels: coordination file-system stale-assigned  (was: coordination file-system)

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a comment updating the community on your progress.  If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone else may work on it.


> Cost too much time to start SourceCoordinator of hdfsFileSource when start JobMaster
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-29617
>                 URL: https://issues.apache.org/jira/browse/FLINK-29617
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / FileSystem, Runtime / Coordination
>    Affects Versions: 1.15.3
>            Reporter: LI Mingkun
>            Assignee: luoyuxia
>            Priority: Major
>              Labels: coordination, file-system, stale-assigned
>         Attachments: image-2022-10-13-19-02-18-555.png, image-2022-10-13-19-02-29-620.png, image-2022-10-13-19-02-35-422.png
>
>
> h1. Scenario:
> Our user use flink batch to compact small files in one day. Flink version : 1.15
> He split pipeline into 24 for each hour. So there are 24 source
>  
> I find it  costs too much time to start SourceCoordinator of hdfsFileSource when start JobMaster
>  
>  as follow:
>  
> !image-2022-10-13-19-02-18-555.png!
>  
> h1. Root Cause:
> I got the root cause after check: 
>  # AbstractFileSource will enumerateSplits when createEnumerator
>  # NotSplittingRecursiveEnumerator need to get fileblockLocation of every fileblock which is a heavy IO operation
>   !image-2022-10-13-19-02-29-620.png!
> !image-2022-10-13-19-02-35-422.png!
>  
> h1. Suggestion
>  # FileSource add option to disable location fetcher
>  # Move location fetcher into IOExecutor



--
This message was sent by Atlassian Jira
(v8.20.10#820010)