[jira] [Resolved] (STORM-3501) Local Cluster worker restarts

["Stig Rohde Døssing (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stig Rohde Døssing resolved STORM-3501.
---------------------------------------
    Fix Version/s: 2.1.0
       Resolution: Fixed

Thanks for the fix [~diogopmonteiro]. Applied to master and 2.1.x-branch.

> Local Cluster worker restarts
> -----------------------------
>
>                 Key: STORM-3501
>                 URL: https://issues.apache.org/jira/browse/STORM-3501
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-server
>    Affects Versions: 2.0.0, 2.1.0
>         Environment: Linux
>            Reporter: Diogo Monteiro
>            Assignee: Diogo Monteiro
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.1.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I was trying to launch a topology that I'm developing (in 2.0.0) and noticed that the worker was getting restarted each ~30 seconds. 
>  I placed a breakpoint in the _kill_ method of _LocalContainer_ ([https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/daemon/supervisor/LocalContainer.java#L66]) to try and understand why the worker was getting restarted. 
>   
>  The call stack was:
> _kill:66, LocalContainer (org.apache.storm.daemon.supervisor)
> killContainerFor:269, Slot (org.apache.storm.daemon.supervisor)
> handleRunning:724, Slot (org.apache.storm.daemon.supervisor) 
> stateMachineStep:218, Slot (org.apache.storm.daemon.supervisor)
> run:931, Slot (org.apache.storm.daemon.supervisor) _
>   
>  With this I can understand that the worker is killed because a blob has changed ([https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/daemon/supervisor/Slot.java#L724]). In fact, there's a changing blob in the _dynamicState_ at that point.
>   
>  I checked the _AsyncLocalizer_ which downloads, caches blobs locally, and notifies the Slot state machine of a changing blob.
>   
>  I noticed this:
>  * [https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L339]
>  * [https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/AsyncLocalizer.java#L265]
>  * [https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedTopologyBlob.java#L142]
>  * [https://github.com/apache/storm/blob/2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae/storm-server/src/main/java/org/apache/storm/localizer/LocallyCachedTopologyBlob.java#L192]
>   
> Which tell me that (correct me if I'm wrong):
>  * Supervisor tries to update blobs each 30 seconds.
>  * The topology jar blob requires extraction of the resources directory (either from a jar or directly in a classpath URL). It does so in _fetchUnzipToTemp_ and it's existence is checked in _isFullyDownloaded_.
>  * The Slot is notified of a changing blob if:
>  * the remote version is different from the local version (the code has changed).
>  * OR the blob is not fully downloaded (the jar exists, and the extracted resources directory exists).
>  
>  Well, I did not have a resources folder under the root of the classpath, and that's why the worker was being restarted each ~30 seconds, as the Slot was being notified of a changing blob everytime _updateBlobs_ ran. 
>  I created a resources folder (with dummy files) under the root of the classpath and the problem is now solved.
>   
>  However, if I understand correctly, the resources folder is only required for _multilang_. Our topologies do not use _multilang_ and this do not happen in Storm 1.1.3 for instance.
>  
> Happy to submit MR.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)