Storm Mock Services

[Apoorv Palkar <ap...@aol.com>]

Hey Dev,


I've been working on the mock services examples for Apache Storm. The three mock services I looked into were: file transfer, job submission, and replication. I got all three to work successfully with each service in its own dummy class. Though the program technically executed, it doesn't mean that Storm should be a correct fit with Airavata. A problem that I see with Storm is that it is fundamentally built to process a stream of data tuples in real time. As such, its internal function nextTuple() is being called indefinitely during the execution of the topology. Judging from the actual Airavata tasks, it seems that we won't receive a continuous stream of data input. This is a major drawback of using Storm. There needs to be extra implementation that is added to fit the Airavata use case. We should aim to fit Storm in as cleanly as possible. This seems to be a small con. Another problem I'm seeming to have keeping track of DAG's that consist of internal DAGs. Currently, we can run a complex/simple DAG in Storm as long as it isn't layered. As Suresh mentioned, if we want to have a multiple layers of DAGs, then we might have to implement this functionality ourselves. Lastly, there isn't an internal mechanism for handling errors. Storm handles basic error cases, but we have to implement more complicated methods for our use case. Other than these three aspects, Storm seems to be doing a good job of execution. We will try to pick between Storm and Helix by the end of this week. You can see via my previous post on Github the execution code of Storm.


* shout out @Marcus for explaining generics/enum use case in Java.