[jira] [Commented] (STORM-2441) Extract client (worker) side artifacts from 'storm-core'

["Jungtaek Lim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950214#comment-15950214 ] 

Jungtaek Lim commented on STORM-2441:
-------------------------------------

Current plan is breaking down 'storm-core' into multiple artifacts:

* 'storm-client'

'storm-client' will contain minimal classes/dependencies needed for submitting topology and executing worker. Supervisor will launch worker with separate classpath which matches 'storm-client'.
So this artifact will be minimum for user side topology.

* 'storm-client-misc' (depends on 'storm-client')

'storm-client-misc' will contain classes and dependencies which are helpful for topology and worker but need to pull additional dependencies. Including this artifact from topology jar is optional.

* 'storm-server' (depends on 'storm-client')

'storm-server' will contain LocalCluster, and related classes (maybe most of daemons). 

* 'storm-core' (depends on 'storm-server')

'storm-core' will contain UI and Logviewer (which are not ported to Java yet), clojure unit tests (and integration tests), and commands, jar transformer (can't we remove this for 2.0.0?). Clojure related dependencies will be kept for this artifact, but can be removed later when porting (including tests) is complete.

* 'storm-webapp' (depends on 'storm-core', renamed from 'storm-drpc-webapp')

''storm-webapp' will contain DRPC http server. Later ported UI and Logviewer will be placed to this artifact.

I'll push the patch for this soon.

> Extract client (worker) side artifacts from 'storm-core'
> --------------------------------------------------------
>
>                 Key: STORM-2441
>                 URL: https://issues.apache.org/jira/browse/STORM-2441
>             Project: Apache Storm
>          Issue Type: Improvement
>          Components: storm-core
>            Reporter: Jungtaek Lim
>            Assignee: Jungtaek Lim
>
> There were some interests for having client side artifacts or modularize Storm.
> - STORM-30
> - STORM-117
> We have been having one huge artifact 'storm-core' for supporting all the things, and recently we extracted 'storm-drpc-server' due to dependency issue.
> We also have been struggling to handle shade & relocation for lots artifacts, but in fact the artifacts which we need to relocate are just Worker related thing. So extracting worker and related classes into separate artifact and assigning different classpath in Storm cluster should help remedying.
> We may still need to struggle with relocation, but the count of target artifacts should be much smaller.
> Another thing we need to consider is LocalCluster, which pulls most of daemons but user topology still need to pull it. So we need to have separate artifact for LocalCluster (maybe also daemons without webapp).
> Last, we need to consider about UI and Logviewer porting. Artifact 'storm-drpc-server' is not generic and I think we can group UI, Logviewer, DRPC (only http server) into this artifact. Artifact name should be changed then.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)