[jira] [Updated] (STORM-2448) Support running workers using older JVMs/storm versions

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/STORM-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim updated STORM-2448:
--------------------------------
    Fix Version/s: 1.2.0

> Support running workers using older JVMs/storm versions
> -------------------------------------------------------
>
>                 Key: STORM-2448
>                 URL: https://issues.apache.org/jira/browse/STORM-2448
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>    Affects Versions: 2.0.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>             Fix For: 2.0.0, 1.0.3, 0.10.3, 1.1.1, 1.2.0
>
>          Time Spent: 3h
>  Remaining Estimate: 0h
>
> As a part of STORM-2441 we are separating out the classpaths for the client+worker process from everything else in storm.  This is great but it really will make some of our users upset, because it is not a rolling upgrade, and because they will need to recompile their topologies (again).
> We have done a really good job in maintaining compatibility with older versions of storm because we use thrift for all communication and state storage.  This means that a new supervisor and or nimbus should be able to talk to just about any existing client/worker out there.  So we should explicitly support this.
> We should add in config options to supervisors.
> {{storm.supported.jvms}} which is a map of the version of the JVM to the JAVA_HOME path for it.
> and
> {{storm.legacy.worker.classpath}} which is a map of the version of storm to a CLASSPATH that can be used to launch a legacy worker process.
> They should be set for all supervisors and nimbus.
> Then we also add in some metadata that the client submits to nimbus along with a topology.  Namely the version of the storm client they are running on and the version of the JVM they are running on.
> Nimbus can then decide (possibly through another config, but probably just through convention with some config overrides) if the version of the client + JVM is compatible with the version of storm + JVM currently on the cluster.  If so it should just let it through.  If not it should pick a version of the JVM + storm that is compatible.  If there are none available it will reject the request.  We should also allow end users to set these configs.
> For this to work well we need some good version matching/sorting code that is lenient, even during rolling upgrades.
> For example if a user submits a topology with a 1.0.3 client to a 2.0.0 cluster. Nimbus sees that it has 1.0.3 installed great it will start running with that, but then we do a rolling upgrade of the cluster and upgrade move to 1.0.4.  The supervisors should be able to launch the workers for that topology with a 1.0.4 classpath.
> As such part of the worker heartbeat should also include the version of storm + JVM that the worker is running with.
> We should display that on the UI and display the version it was submitted with on the UI too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)