You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Karthikeyan Muthukumar <mk...@gmail.com> on 2016/03/07 21:46:51 UTC
Updating reference data once a day in Spark Streaming job
Hi,
We have reference data pulled in from an RDBMS through a Sqoop job, this
reference data is pulled into the Analytics platform once a day.
We have a Spark Streaming job, where at job bootup we read the reference
data, and then join this reference data with continuously flowing event
data. When the reference data gets updated once a day, how do I make sure
the Spark Streaming job uses the newly updated reference data?
One simple way is to bounce the Spark Streaming job once a day after new
Reference data is imported but is there a better & less-destructive
approach?
Thanks & Regards
MK