Defining topologies at runtime (inspired by functional logic programming)

[Joël Kuiper <me...@joelkuiper.eu>]

Hey, 

So I’m just philosophising a bit here: is it possible to create and submit topologies at runtime? 

The thought here is to have a set of triples that define which operations are possible on the data at hand. 
A user can then request a certain end-result, and by using logic programming (or SPARQL) the graph for getting that result from the input is constructed dynamically from the triples.
Generating the Storm topology would then involve building/retrieving the appropriate bolts and hooking them up in the manner described by the query. 
Then the topology could be submitted, the job processed and the result returned; after which the topology could be discarded. 

It seems like a very nice way of creating (efficient) transformations on data to many output formats. 

The concrete application here would be bioinformatics/epidemiology where a lot of different data formats, sources and tools exists; but how to wire them is often ad-hoc. 
Using logic programming to find the paths and getting storm to automatically build and run the topology would seem nice (albeit a bit academic)! 

Any thoughts on this?

Re: Defining topologies at runtime (inspired by functional logic programming)

[Alexei Osipov <al...@caesarsint.com>]

Yes, it's already possible.

1) You can build topology dynamically from your java code.
2) You can upload a jar with all dependencies of your bolts to Nimbus. 
This can be done with StormSubmitter.submitJar(Map conf, String localJar)
3) You can submit your topology with NimbusClient. Check 
StormSubmitter.submitTopology(...) for example code.
4) For querying topology check DRPC API.

Best regards,
Alexei Osipov.

On 18.04.2014 17:48, Joël Kuiper wrote:
> Hey,
>
> So I’m just philosophising a bit here: is it possible to create and submit topologies at runtime?
>
> The thought here is to have a set of triples that define which operations are possible on the data at hand.
> A user can then request a certain end-result, and by using logic programming (or SPARQL) the graph for getting that result from the input is constructed dynamically from the triples.
> Generating the Storm topology would then involve building/retrieving the appropriate bolts and hooking them up in the manner described by the query.
> Then the topology could be submitted, the job processed and the result returned; after which the topology could be discarded.
>
> It seems like a very nice way of creating (efficient) transformations on data to many output formats.
>
> The concrete application here would be bioinformatics/epidemiology where a lot of different data formats, sources and tools exists; but how to wire them is often ad-hoc.
> Using logic programming to find the paths and getting storm to automatically build and run the topology would seem nice (albeit a bit academic)!
>
> Any thoughts on this?
>
>
>