Spark Streaming. Real-time save data and visualize on dashboard

[tencas <di...@gmail.com>]

I've developed an application using Apache Spark Streaming, that reads simple
info from plane sensors like acceleration, via TCP sockets on json format,
and analyse it.

I'd like to be able to persist this info from each "flight" on real-time,
while it is shown on any responsive dashboard.

I just don't know if is it possible to use a no-SQL database like Mongo,
Cassandra in collaboration with a monitoring tools like Grafana,Kabana, and
it make sense.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Real-time-save-data-and-visualize-on-dashboard-tp28587.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Spark Streaming. Real-time save data and visualize on dashboard

[Pierce Lamb <ri...@gmail.com>]

Hi,

It is possible to use Mongo or Cassandra to persist results from Spark. In
fact, a wide variety of data stores are available to use with Spark and
many are aimed at serving queries for dashboard visualizations. I cannot
comment on which work well with Grafana or Kabana, however, I've listed
(with links) a majority of the data stores that have an existing connector
or integration with Spark here:

http://stackoverflow.com/a/39753976/3723346

Hope this helps,

Pierce

On Tue, Apr 11, 2017 at 7:35 AM, tencas <di...@gmail.com> wrote:

> I've developed an application using Apache Spark Streaming, that reads
> simple
> info from plane sensors like acceleration, via TCP sockets on json format,
> and analyse it.
>
> I'd like to be able to persist this info from each "flight" on real-time,
> while it is shown on any responsive dashboard.
>
> I just don't know if is it possible to use a no-SQL database like Mongo,
> Cassandra in collaboration with a monitoring tools like Grafana,Kabana, and
> it make sense.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Spark-Streaming-Real-time-save-data-
> and-visualize-on-dashboard-tp28587.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: Spark Streaming. Real-time save data and visualize on dashboard

[Sam Elamin <hu...@gmail.com>]

Hi

To be honest there are a variety of options but it all comes down to who
will be querying these dashboards.

If the end user is an engineer then the ELK stack is fine and I can attest
to the ease of use of kibana since I used it quite heavily.

On the other hand in my experience it isnt the engineers that are in charge
of reporting so if the end user is a data analyst or data scientist then
they are most comfortable using SQL and would be slightly aversed to
learning the nuances of creating dashboards and using elastic search. Trust
me no matter how much you try, these folks are more comfortable using sql
and Tableau like platforms. So you will have to educate them, Not to
mention the fact that any new hire will have to undergo the same training
to be productive

My suggestion for that is to push your data to Google BigQuery
<https://cloud.google.com/bigquery/>. It really is simple to use and people
can just focus on writing their queries. It also returns within seconds for
queries over terabytes of data. The caveat here is that you are paying per
query. But it's $5 for 1 TB which is peanuts really. Its a managed service
so there is zero setup costs and management compared to the other services.
I suppose in the end you are paying to abstract that knowledge away

Happy to answer any questions you might have

Kind Regards
Sam




On Wed, 12 Apr 2017 at 09:36, tencas <di...@gmail.com> wrote:

> Hi Gaurav1809 ,
>
> I was thinking about using elasticsearch + kibana too (actually don't know
> the differences between ELK and elasticsearch).
> I was wondering about pros and cons of using a document indexer vs NoSQL
> database.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Spark-Streaming-Real-time-save-data-
> and-visualize-on-dashboard-tp28587p28589.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Re: Spark Streaming. Real-time save data and visualize on dashboard

[tencas <di...@gmail.com>]

Hi Gaurav1809 ,

I was thinking about using elasticsearch + kibana too (actually don't know
the differences between ELK and elasticsearch).
I was wondering about pros and cons of using a document indexer vs NoSQL
database.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Real-time-save-data-and-visualize-on-dashboard-tp28587p28589.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Spark Streaming. Real-time save data and visualize on dashboard

[Gaurav1809 <ga...@gmail.com>]

May be you can injest your data in ELK and use Kibana for live reporting. Of
course there can be better way of doing this. Waiting for others to share
their opinion. Thanks.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Real-time-save-data-and-visualize-on-dashboard-tp28587p28588.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org