You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Chang Chen <ba...@gmail.com> on 2017/11/20 12:00:25 UTC
How to print plan of Structured Streaming DataFrame
Hi Guys
I modified StructuredNetworkWordCount to see what the executed plan is,
here are my codes:
val wordCounts = words.groupBy("value").count()
// Start running the query that prints the running counts to the console
val query = wordCounts.writeStream
.outputMode("complete")
.format("console")
.start()
wordCounts.explain() // additional codes
But it failed with “AnalysisException: Queries with streaming sources must
be executed with writeStream.start()”?
Thanks
Chang
Re: How to print plan of Structured Streaming DataFrame
Posted by Chang Chen <ba...@gmail.com>.
Yes, I switched to query.explain, though it did't fail, I still can't make
it print executed plan, is there any example?
I first try to placed query.explain() exactly after start, but spark
reports no data.
Hera are my codes:
while (query.awaitTermination(20000)) { // 1
query.explain() // 2
}
query.explain() // 3
The interesting part is query.awaitTermination(20000) will terminate
stream engine, so query.explain() at line 2 never be executed.
On Tue, Nov 21, 2017 at 1:49 PM, Liang-Chi Hsieh <vi...@gmail.com> wrote:
>
> wordCounts.explain() -> query.explain()?
>
>
> Chang Chen wrote
> > Hi Guys
> >
> > I modified StructuredNetworkWordCount to see what the executed plan is,
> > here are my codes:
> >
> > val wordCounts = words.groupBy("value").count()
> >
> > // Start running the query that prints the running counts to the console
> > val query = wordCounts.writeStream
> > .outputMode("complete")
> > .format("console")
> > .start()
> >
> > wordCounts.explain() // additional codes
> >
> >
> > But it failed with “AnalysisException: Queries with streaming sources
> must
> > be executed with writeStream.start()”?
> >
> >
> > Thanks
> > Chang
>
>
>
>
>
> -----
> Liang-Chi Hsieh | @viirya
> Spark Technology Center
> http://www.spark.tc/
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: How to print plan of Structured Streaming DataFrame
Posted by Liang-Chi Hsieh <vi...@gmail.com>.
wordCounts.explain() -> query.explain()?
Chang Chen wrote
> Hi Guys
>
> I modified StructuredNetworkWordCount to see what the executed plan is,
> here are my codes:
>
> val wordCounts = words.groupBy("value").count()
>
> // Start running the query that prints the running counts to the console
> val query = wordCounts.writeStream
> .outputMode("complete")
> .format("console")
> .start()
>
> wordCounts.explain() // additional codes
>
>
> But it failed with “AnalysisException: Queries with streaming sources must
> be executed with writeStream.start()”?
>
>
> Thanks
> Chang
-----
Liang-Chi Hsieh | @viirya
Spark Technology Center
http://www.spark.tc/
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: How to print plan of Structured Streaming DataFrame
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
-dev +user
Which Spark version are you using? There is a bug in the old Spark. Try to
use the latest version.
In addition, you can call `query.explain()` as well.
On Mon, Nov 20, 2017 at 4:00 AM, Chang Chen <ba...@gmail.com> wrote:
> Hi Guys
>
> I modified StructuredNetworkWordCount to see what the executed plan is,
> here are my codes:
>
> val wordCounts = words.groupBy("value").count()
>
> // Start running the query that prints the running counts to the console
> val query = wordCounts.writeStream
> .outputMode("complete")
> .format("console")
> .start()
>
> wordCounts.explain() // additional codes
>
>
> But it failed with “AnalysisException: Queries with streaming sources must
> be executed with writeStream.start()”?
>
>
> Thanks
> Chang
>
>