You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/09/12 21:43:13 UTC
Zeppelin patterns with the streaming data
The latest version of Zeppelin 0.6.1 looks pretty impressive with Spark 2
and also with Spark Thrift server (it runs on Hive Thrift server) and uses
Hive execution engine. Make sure that you do not use MapReduce as Hive's
execution engine.
Now for streaming data (in this case some test generated data using Kafka
topic), I store them as text file on HDFS. To my surprise text files (all
created every two seconds under some HDFS some directory) with timestamp
added to the file name seem to be pretty fast. I am sceptical what benefit
one gets if I decide to store them as Hbase file? Anyone can shed some
light on it?
Also I use Zeppelin to do some plots on the stored security prices. These
are all Fictitious (the data belonging to these securities are all
fictitious and randomly generated). Now I don't think there is anyway one
can automate Zeppelin to run the same code say every x seconds? Thinking
loud the other alternative is to add new data as comes in and tail off the
old data? Has anyone done any work of this type?
Anyway I show you a sample graph below. Appreciate any ideas. My view is
that Zeppelin is not designed for real time dashboard but increasingly
looking good and may be with some change one can use it near real time?
[image: Inline images 2]
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Zeppelin patterns with the streaming data
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Chanh,
Yes indeed. Apparently it is implemented through a class of its own. I have
specified a refresh of every 15 seconds.
Obviously if there is an issue then the cron will not be able to refresh
but you cannot sort out that problem from the web page anyway
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 13 September 2016 at 09:00, Chanh Le <gi...@gmail.com> wrote:
> Hi Mich,
> I think it can http://www.quartz-scheduler.org/documentation/
> quartz-2.1.x/tutorials/crontrigger
>
>
>
>
> On Sep 13, 2016, at 1:57 PM, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
> Thanks Sachin.
>
> The cron gives the granularity of 1 min. On normal one can use wait 10 and
> loop in the cron to run the job every 10 seconds. I am not sure that is
> possible with Zeppelin?
>
> cheers
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 13 September 2016 at 05:05, Sachin Janani <sj...@snappydata.io>
> wrote:
>
>> Good to see that you are enjoying zeppelin.You can schedule the paragraph
>> running after every x seconds.You can find this option on top of the
>> notebook just beside delete notebook button.
>>
>>
>> Regards,
>> Sachin Janani
>> http://snappydatainc.github.io/snappydata/
>>
>>
>> On Tue, Sep 13, 2016 at 3:13 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> The latest version of Zeppelin 0.6.1 looks pretty impressive with Spark
>>> 2 and also with Spark Thrift server (it runs on Hive Thrift server) and
>>> uses Hive execution engine. Make sure that you do not use MapReduce as
>>> Hive's execution engine.
>>>
>>> Now for streaming data (in this case some test generated data using
>>> Kafka topic), I store them as text file on HDFS. To my surprise text files
>>> (all created every two seconds under some HDFS some directory) with
>>> timestamp added to the file name seem to be pretty fast. I am sceptical
>>> what benefit one gets if I decide to store them as Hbase file? Anyone can
>>> shed some light on it?
>>>
>>> Also I use Zeppelin to do some plots on the stored security prices.
>>> These are all Fictitious (the data belonging to these securities are all
>>> fictitious and randomly generated). Now I don't think there is anyway one
>>> can automate Zeppelin to run the same code say every x seconds? Thinking
>>> loud the other alternative is to add new data as comes in and tail off the
>>> old data? Has anyone done any work of this type?
>>>
>>> Anyway I show you a sample graph below. Appreciate any ideas. My view is
>>> that Zeppelin is not designed for real time dashboard but increasingly
>>> looking good and may be with some change one can use it near real time?
>>>
>>>
>>> <image.png>
>>>
>>> Thanks
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>
>>
>
>
Re: Zeppelin patterns with the streaming data
Posted by Chanh Le <gi...@gmail.com>.
Hi Mich,
I think it can http://www.quartz-scheduler.org/documentation/quartz-2.1.x/tutorials/crontrigger <http://www.quartz-scheduler.org/documentation/quartz-2.1.x/tutorials/crontrigger>
> On Sep 13, 2016, at 1:57 PM, Mich Talebzadeh <mi...@gmail.com> wrote:
>
> Thanks Sachin.
>
> The cron gives the granularity of 1 min. On normal one can use wait 10 and loop in the cron to run the job every 10 seconds. I am not sure that is possible with Zeppelin?
>
> cheers
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
>
>
> On 13 September 2016 at 05:05, Sachin Janani <sjanani@snappydata.io <ma...@snappydata.io>> wrote:
> Good to see that you are enjoying zeppelin.You can schedule the paragraph running after every x seconds.You can find this option on top of the notebook just beside delete notebook button.
>
>
> Regards,
> Sachin Janani
> http://snappydatainc.github.io/snappydata/ <http://snappydatainc.github.io/snappydata/>
>
>
> On Tue, Sep 13, 2016 at 3:13 AM, Mich Talebzadeh <mich.talebzadeh@gmail.com <ma...@gmail.com>> wrote:
> The latest version of Zeppelin 0.6.1 looks pretty impressive with Spark 2 and also with Spark Thrift server (it runs on Hive Thrift server) and uses Hive execution engine. Make sure that you do not use MapReduce as Hive's execution engine.
>
> Now for streaming data (in this case some test generated data using Kafka topic), I store them as text file on HDFS. To my surprise text files (all created every two seconds under some HDFS some directory) with timestamp added to the file name seem to be pretty fast. I am sceptical what benefit one gets if I decide to store them as Hbase file? Anyone can shed some light on it?
>
> Also I use Zeppelin to do some plots on the stored security prices. These are all Fictitious (the data belonging to these securities are all fictitious and randomly generated). Now I don't think there is anyway one can automate Zeppelin to run the same code say every x seconds? Thinking loud the other alternative is to add new data as comes in and tail off the old data? Has anyone done any work of this type?
>
> Anyway I show you a sample graph below. Appreciate any ideas. My view is that Zeppelin is not designed for real time dashboard but increasingly looking good and may be with some change one can use it near real time?
>
>
> <image.png>
>
> Thanks
>
> Dr Mich Talebzadeh
>
> LinkedIn https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>
>
> http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/>
>
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
>
>
>
Re: Zeppelin patterns with the streaming data
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Sachin.
The cron gives the granularity of 1 min. On normal one can use wait 10 and
loop in the cron to run the job every 10 seconds. I am not sure that is
possible with Zeppelin?
cheers
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 13 September 2016 at 05:05, Sachin Janani <sj...@snappydata.io> wrote:
> Good to see that you are enjoying zeppelin.You can schedule the paragraph
> running after every x seconds.You can find this option on top of the
> notebook just beside delete notebook button.
>
>
> Regards,
> Sachin Janani
> http://snappydatainc.github.io/snappydata/
>
>
> On Tue, Sep 13, 2016 at 3:13 AM, Mich Talebzadeh <
> mich.talebzadeh@gmail.com> wrote:
>
>> The latest version of Zeppelin 0.6.1 looks pretty impressive with Spark 2
>> and also with Spark Thrift server (it runs on Hive Thrift server) and uses
>> Hive execution engine. Make sure that you do not use MapReduce as Hive's
>> execution engine.
>>
>> Now for streaming data (in this case some test generated data using Kafka
>> topic), I store them as text file on HDFS. To my surprise text files (all
>> created every two seconds under some HDFS some directory) with timestamp
>> added to the file name seem to be pretty fast. I am sceptical what benefit
>> one gets if I decide to store them as Hbase file? Anyone can shed some
>> light on it?
>>
>> Also I use Zeppelin to do some plots on the stored security prices. These
>> are all Fictitious (the data belonging to these securities are all
>> fictitious and randomly generated). Now I don't think there is anyway one
>> can automate Zeppelin to run the same code say every x seconds? Thinking
>> loud the other alternative is to add new data as comes in and tail off the
>> old data? Has anyone done any work of this type?
>>
>> Anyway I show you a sample graph below. Appreciate any ideas. My view is
>> that Zeppelin is not designed for real time dashboard but increasingly
>> looking good and may be with some change one can use it near real time?
>>
>>
>> [image: Inline images 2]
>>
>>
>> Thanks
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
>
Re: Zeppelin patterns with the streaming data
Posted by Sachin Janani <sj...@snappydata.io>.
Good to see that you are enjoying zeppelin.You can schedule the paragraph
running after every x seconds.You can find this option on top of the
notebook just beside delete notebook button.
Regards,
Sachin Janani
http://snappydatainc.github.io/snappydata/
On Tue, Sep 13, 2016 at 3:13 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> The latest version of Zeppelin 0.6.1 looks pretty impressive with Spark 2
> and also with Spark Thrift server (it runs on Hive Thrift server) and uses
> Hive execution engine. Make sure that you do not use MapReduce as Hive's
> execution engine.
>
> Now for streaming data (in this case some test generated data using Kafka
> topic), I store them as text file on HDFS. To my surprise text files (all
> created every two seconds under some HDFS some directory) with timestamp
> added to the file name seem to be pretty fast. I am sceptical what benefit
> one gets if I decide to store them as Hbase file? Anyone can shed some
> light on it?
>
> Also I use Zeppelin to do some plots on the stored security prices. These
> are all Fictitious (the data belonging to these securities are all
> fictitious and randomly generated). Now I don't think there is anyway one
> can automate Zeppelin to run the same code say every x seconds? Thinking
> loud the other alternative is to add new data as comes in and tail off the
> old data? Has anyone done any work of this type?
>
> Anyway I show you a sample graph below. Appreciate any ideas. My view is
> that Zeppelin is not designed for real time dashboard but increasingly
> looking good and may be with some change one can use it near real time?
>
>
> [image: Inline images 2]
>
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>