Realtime counting job with reading access from flatmappers

[Dirk Weissenborn <di...@gmail.com>]

Hey guys,

first of all, nice job with Spark ;)

I want to you Spark in the following setting and I am not completely sure
what the best architecture would be. This is why I would like to ask for
your opinion.

Job:

- read object from input stream
- input is a set of ids
- map input ids to new ids using real time counts for the input-  ids of
the objects from a counting store (e.g., key value store)
- increment counts of the new ids and decrement count of the old ids in the
counting store


Now I am not completely sure, which kind of store I should use. The stores
are too big to get serialized to all workers, rather I'd like to have it
somewhere centralized and accessible for each worker. I think an easy
solution would be to create a connection to an (in-memory) database (e.g.,
memcached) and open a connection in each mapper. But I am not sure how I
can do that, because in the map function a connection would be established
for each incoming object, which would be a big overhead. So my question is:
how can I setup a connection to a db in a mapper without having to create
it for each incoming object? Or is there even a simpler solution to my job
description?

I appreciate your help.

Best,
Dirk