Storm Topology/Jar Submission

[Bryan Baugher <bj...@gmail.com>]

Hi everyone,

For some background, I am using Storm 0.8.2 and submitting/starting storm
topologies from a web service using a modified version of the
StormSubmitter[1]. The initial implementation ended up uploading the
topology jar every time. I started to look into ways to reduce the number
of uploads and saw that nimbus has a thread that cleans out its inbox based
on a jar's age (default 1hr). So I have played around with ideas like
deploying the jar on nimbus in some known location that won't get cleaned,
caching the path to where we uploaded the jar and expiring it the same time
nimbus would, or just continuing to upload every time and setting the
cleaning to be more aggressive but no particular option stood out to me.

So I have a few questions,

   - Am I doing anything wrong or are there things I can be doing better or
   differently?
   - Why is the default behavior to clean a jar from the nimbus inbox 1hr?
   I would think the nimbus wouldn't need this for very long at all.
   - If some nodes die and the topology is shuffled around how do the new
   nodes get the jar for the topology assuming 1hr has passed and the jar has
   been cleaned from the nimbus inbox?

[1] -
https://github.com/nathanmarz/storm/blob/moved-to-apache/storm-core/src/jvm/backtype/storm/StormSubmitter.java

-Bryan