Does Zeppelin 0.7.1 work with Spark Structured Streaming 2.1.1?

[kant kodali <ka...@gmail.com>]

Hi All,

I have the following code

StreamingQuery query = df2.writeStream().outputMode("complete").queryName(
"foo").option("truncate", "false").format("console").start();

query.awaitTermination();


and it works fine however when I change it to the below code. I do get the
output but only once and I tried running *%spark.sql select * from foo* over
and over again but I don't see results getting updated but in console
format like above it works perfect fine I can see updates on each batch so
should I be doing something else for memory sink?

StreamingQuery query =
df2.writeStream().outputMode("complete").queryName("foo").option("truncate",
"false").format("memory").start();

%spark.sql select * from foo


Thanks!

Re: Does Zeppelin 0.7.1 work with Spark Structured Streaming 2.1.1?

[Felix Cheung <fe...@hotmail.com>]

I think you can have one notebook for setup - calling the source and sink as you have below; then another notebook to run the %sql to read from memory temp view and output to a visualization.

This way you could have the 2nd notebook be a dashboard and run (i.e. refresh automatically) on a schedule.

As for
df2.writeStream().outputMode("complete").queryName("foo").option("truncate","false").format("console").start();

You can't use queryName with format console - this is not supported by Spark Structured Streaming.

_____________________________
From: kant kodali <ka...@gmail.com>>
Sent: Friday, May 19, 2017 10:29 AM
Subject: Re: Does Zeppelin 0.7.1 work with Spark Structured Streaming 2.1.1?
To: <us...@zeppelin.apache.org>>


Look for something that is more like in this video where the graphs automatically update themselves. Is that possible in Zeppelin?

https://www.youtube.com/watch?v=IJmFTXvUZgY

You can watch it from 9:20


On Fri, May 19, 2017 at 9:21 AM, kant kodali <ka...@gmail.com>> wrote:
Hi All,

I have the following code

StreamingQuery query = df2.writeStream().outputMode("complete").queryName("foo").option("truncate","false").format("console").start();

query.awaitTermination();

and it works fine however when I change it to the below code. I do get the output but only once and I tried running %spark.sql select * from foo over and over again but I don't see results getting updated but in console format like above it works perfect fine I can see updates on each batch so should I be doing something else for memory sink?


StreamingQuery query = df2.writeStream().outputMode("complete").queryName("foo").option("truncate", "false").format("memory").start();

%spark.sql select * from foo


Thanks!

Re: Does Zeppelin 0.7.1 work with Spark Structured Streaming 2.1.1?

[kant kodali <ka...@gmail.com>]

Look for something that is more like in this video where the graphs
automatically update themselves. Is that possible in Zeppelin?

https://www.youtube.com/watch?v=IJmFTXvUZgY

You can watch it from 9:20


On Fri, May 19, 2017 at 9:21 AM, kant kodali <ka...@gmail.com> wrote:

> Hi All,
>
> I have the following code
>
> StreamingQuery query = df2.writeStream().outputMode("complete").queryName(
> "foo").option("truncate", "false").format("console").start();
>
> query.awaitTermination();
>
>
> and it works fine however when I change it to the below code. I do get the
> output but only once and I tried running *%spark.sql select * from foo* over
> and over again but I don't see results getting updated but in console
> format like above it works perfect fine I can see updates on each batch so
> should I be doing something else for memory sink?
>
> StreamingQuery query = df2.writeStream().outputMode("complete").queryName("foo").option("truncate", "false").format("memory").start();
>
> %spark.sql select * from foo
>
>
> Thanks!
>
>
>