You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Chang Chen <ba...@gmail.com> on 2017/11/20 12:00:25 UTC

How to print plan of Structured Streaming DataFrame

Hi Guys

I modified StructuredNetworkWordCount to see what the executed plan is,
here are my codes:

val wordCounts = words.groupBy("value").count()

// Start running the query that prints the running counts to the console
val query = wordCounts.writeStream
  .outputMode("complete")
  .format("console")
  .start()

wordCounts.explain()  // additional codes


But it failed with “AnalysisException: Queries with streaming sources must
be executed with writeStream.start()”?


Thanks
Chang

Re: How to print plan of Structured Streaming DataFrame

Posted by Chang Chen <ba...@gmail.com>.
Yes, I switched to query.explain, though it did't fail,  I still can't make
it print executed plan, is there any example?

I first try to placed query.explain() exactly after start, but spark
reports no data.

Hera are my codes:

while (query.awaitTermination(20000)) {  // 1
  query.explain()                                       // 2
}
query.explain()                                        // 3

The interesting part is  query.awaitTermination(20000) will terminate
stream engine, so query.explain() at line 2 never be executed.


On Tue, Nov 21, 2017 at 1:49 PM, Liang-Chi Hsieh <vi...@gmail.com> wrote:

>
> wordCounts.explain() -> query.explain()?
>
>
> Chang Chen wrote
> > Hi Guys
> >
> > I modified StructuredNetworkWordCount to see what the executed plan is,
> > here are my codes:
> >
> > val wordCounts = words.groupBy("value").count()
> >
> > // Start running the query that prints the running counts to the console
> > val query = wordCounts.writeStream
> >   .outputMode("complete")
> >   .format("console")
> >   .start()
> >
> > wordCounts.explain()  // additional codes
> >
> >
> > But it failed with “AnalysisException: Queries with streaming sources
> must
> > be executed with writeStream.start()”?
> >
> >
> > Thanks
> > Chang
>
>
>
>
>
> -----
> Liang-Chi Hsieh | @viirya
> Spark Technology Center
> http://www.spark.tc/
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>

Re: How to print plan of Structured Streaming DataFrame

Posted by Liang-Chi Hsieh <vi...@gmail.com>.
wordCounts.explain() -> query.explain()?


Chang Chen wrote
> Hi Guys
> 
> I modified StructuredNetworkWordCount to see what the executed plan is,
> here are my codes:
> 
> val wordCounts = words.groupBy("value").count()
> 
> // Start running the query that prints the running counts to the console
> val query = wordCounts.writeStream
>   .outputMode("complete")
>   .format("console")
>   .start()
> 
> wordCounts.explain()  // additional codes
> 
> 
> But it failed with “AnalysisException: Queries with streaming sources must
> be executed with writeStream.start()”?
> 
> 
> Thanks
> Chang





-----
Liang-Chi Hsieh | @viirya 
Spark Technology Center 
http://www.spark.tc/ 
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Re: How to print plan of Structured Streaming DataFrame

Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
-dev +user

Which Spark version are you using? There is a bug in the old Spark. Try to
use the latest version.

In addition, you can call `query.explain()` as well.

On Mon, Nov 20, 2017 at 4:00 AM, Chang Chen <ba...@gmail.com> wrote:

> Hi Guys
>
> I modified StructuredNetworkWordCount to see what the executed plan is,
> here are my codes:
>
> val wordCounts = words.groupBy("value").count()
>
> // Start running the query that prints the running counts to the console
> val query = wordCounts.writeStream
>   .outputMode("complete")
>   .format("console")
>   .start()
>
> wordCounts.explain()  // additional codes
>
>
> But it failed with “AnalysisException: Queries with streaming sources must
> be executed with writeStream.start()”?
>
>
> Thanks
> Chang
>
>