random idea

[kant kodali <ka...@gmail.com>]

Hi Guys,

I have a random idea and it would be great to receive some input.

Can we have a HTTP2 Based receiver for Spark Streaming? I am wondering why
not build micro services using Spark when needed? I can see it is not meant
for that but I like to think it can be possible. To be more concrete, here
is a scenario.

It is a very common pattern in the industry that people build a service
layer on top of a database (say Cassandra, HBASE or whatever). So all the
service layer does is expose few service functions for the application via
HTTP2 or RPC or some other protocol and the service layer is built simply
by using the database driver underneath. After some point people will start
thinking how to scale this service layer so they start introducing
components service discovery, resource management and so on.  I feel like I
can get all this from Spark with added bonus such as leveraging the
locality using the connectors written for a distributed database. Now I
understand Spark Streaming may not be ready for this given its latency and
mini-batch model but at some point I am confident that this latency issue
will be addressed in which case I do see a path for this. what do you think?

Thanks!
kant