[jira] [Commented] (STORM-2422) Serialized Trident topology size does not grow linerarly

["Robert Joseph Evans (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932614#comment-15932614 ] 

Robert Joseph Evans commented on STORM-2422:
--------------------------------------------

[~kabhwan],

The only real thing to be careful with pulling it into another branch is making sure to set the serialVersionUID correctly.  I tested it for 2.x and I know that it works.  I think that 1.x will have the same ID as 2.x, but because we were not specifying it before I really would want to check before back-porting.

> Serialized Trident topology size does not grow linerarly
> --------------------------------------------------------
>
>                 Key: STORM-2422
>                 URL: https://issues.apache.org/jira/browse/STORM-2422
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: trident
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>             Fix For: 2.0.0
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Many of the Trident bolts contain a serialized version of the entire trident graph.  This means that as the size of the graph grows, and the number of bolts grows with it, that the serialized topology grows quadraticly.
> The bolt only uses a small portion of that graph so we can filter out most of that.
> As an extreme example a topology with 1000 Bolts (4000 nodes in the graph) can grow to be over 1GB in size.  If we trim out the parts of the graph that are not needed the serialized topology is < 40MB in size.
> This is a backwards compatible change, because we are only stripping out data that is never touched by the bolt anyways.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)