You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by kant kodali <ka...@gmail.com> on 2016/11/30 08:47:29 UTC

Can I have two different receivers for my Spark client program?

HI All,

I am wondering if it makes sense to have two receivers inside my Spark
Client program?

The use case is as follows.

1) We have to support a feed from Kafka so this will be a direct receiver
#1. We need to perform batch inserts from kafka feed to Cassandra.

2) an gRPC receiver where we get a RPC request or HTTP request (since grpc
uses HTTP2) to retrieve a record  or perform a table scan or select a good
chunk of records from Cassandra. so this is more like a microservice with
spark streaming layer underneath where some of the requests such as batch
reads and batch inserts will go through spark.

Any feedback or thoughts will be great!

Thanks!