You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@storm.apache.org by "Ethan Li (JIRA)" <ji...@apache.org> on 2019/08/10 22:24:00 UTC

[jira] [Updated] (STORM-3396) uploading dependency jars too slow when StormSubmitter and Nimbus located in different IDC

     [ https://issues.apache.org/jira/browse/STORM-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Li updated STORM-3396:
----------------------------
    Fix Version/s:     (was: 2.0.1)
                   2.1.0

> uploading dependency jars too slow when StormSubmitter and Nimbus located in different IDC
> ------------------------------------------------------------------------------------------
>
>                 Key: STORM-3396
>                 URL: https://issues.apache.org/jira/browse/STORM-3396
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>    Affects Versions: 2.0.0, 1.2.2
>            Reporter: StaticMian
>            Assignee: StaticMian
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.1.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
>      when storm client and server is locating in different IDC(one is in Beijing, while another in Shanghai), uploading dependency jars may take a very long long time(in my case, 31minutes!)...
>     when I digged into this, I found that in DependencyUploader,  method "uploadDependencyToBlobStore" using JDK NIO's Files.copy to upload local jars to remote Blob server. In Files.copy(Path, OutputStream), the buffer size is 8k by default, given that latency between Beijing and Shanghai is about 20ms, a dependency fat jar of 360M finally cost me 'a lunch time' to finish uploading!!!
>    



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)