You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by GitBox <gi...@apache.org> on 2022/05/18 04:21:21 UTC
[GitHub] [spark-website] HeartSaVioR opened a new pull request, #389: Promote Structured Streaming over Spark Streaming (DStream)
HeartSaVioR opened a new pull request, #389:
URL: https://github.com/apache/spark-website/pull/389
This PR proposes to promote Structured Streaming over Spark Streaming, as we see efforts of community are more focused on Structured Streaming (based on Spark SQL) than Spark Streaming (DStream). We would like to encourage end users to use Structured Streaming than Spark Streaming whenever possible for their workloads.
Here are screenshots of pages this PR changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #389:
URL: https://github.com/apache/spark-website/pull/389#issuecomment-1130910779
Thanks everyone for reviewing! I'll merge this once the build passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] gengliangwang commented on a diff in pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #389:
URL: https://github.com/apache/spark-website/pull/389#discussion_r875569794
##########
site/streaming/index.html:
##########
@@ -125,135 +125,98 @@
<div class="row mt-4">
<div class="col-12 col-md-9">
<div class="jumbotron">
- <b>Spark streaming</b> makes it easy to build scalable fault-tolerant streaming
- applications.
+ <a href="/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.
</div>
<div class="row row-padded">
<div class="col-md-7 col-sm-7">
- <h2>Ease of use</h2>
- <p class="lead">
- Build applications through high-level operators.
- </p>
+ <h2>Ease to use</h2>
<p>
- Spark Streaming brings Apache Spark's
- <a href="/docs/latest/streaming-programming-guide.html">language-integrated API</a>
- to stream processing, letting you write streaming jobs the same way you write batch jobs.
- It supports Java, Scala and Python.
+ Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks
+ so that you can build streaming applications and pipelines without learning any new concepts or tools.
</p>
</div>
<div class="col-md-5 col-sm-5 col-padded-top col-center">
<div style="margin-top: 15px; text-align: left; display: inline-block;">
<div class="code">
- TwitterUtils.createStream(...)<br />
- .<span class="sparkop">filter</span>(<span class="closure">_.getText.contains("Spark")</span>)<br />
- .<span class="sparkop">countByWindow</span>(Seconds(5))
+ spark<br />
+ .<span class="sparkop">readStream</span><br />
+ .<span class="sparkop">select</span>(<span class="closure">cast("string").alias("jsonData")</span>)<br />
Review Comment:
BTW, shall we put the `<span class="closure">` around the double quotes? It will look more like the common highlight style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on pull request #389: Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #389:
URL: https://github.com/apache/spark-website/pull/389#issuecomment-1129550667
cc. @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] gengliangwang commented on a diff in pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on code in PR #389:
URL: https://github.com/apache/spark-website/pull/389#discussion_r875567937
##########
site/streaming/index.html:
##########
@@ -125,135 +125,98 @@
<div class="row mt-4">
<div class="col-12 col-md-9">
<div class="jumbotron">
- <b>Spark streaming</b> makes it easy to build scalable fault-tolerant streaming
- applications.
+ <a href="/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.
</div>
<div class="row row-padded">
<div class="col-md-7 col-sm-7">
- <h2>Ease of use</h2>
- <p class="lead">
- Build applications through high-level operators.
- </p>
+ <h2>Ease to use</h2>
<p>
- Spark Streaming brings Apache Spark's
- <a href="/docs/latest/streaming-programming-guide.html">language-integrated API</a>
- to stream processing, letting you write streaming jobs the same way you write batch jobs.
- It supports Java, Scala and Python.
+ Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks
+ so that you can build streaming applications and pipelines without learning any new concepts or tools.
</p>
</div>
<div class="col-md-5 col-sm-5 col-padded-top col-center">
<div style="margin-top: 15px; text-align: left; display: inline-block;">
<div class="code">
- TwitterUtils.createStream(...)<br />
- .<span class="sparkop">filter</span>(<span class="closure">_.getText.contains("Spark")</span>)<br />
- .<span class="sparkop">countByWindow</span>(Seconds(5))
+ spark<br />
+ .<span class="sparkop">readStream</span><br />
+ .<span class="sparkop">select</span>(<span class="closure">cast("string").alias("jsonData")</span>)<br />
Review Comment:
nit: cast => column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on PR #389:
URL: https://github.com/apache/spark-website/pull/389#issuecomment-1131056571
Thanks @gengliangwang for merging this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on a diff in pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #389:
URL: https://github.com/apache/spark-website/pull/389#discussion_r875629915
##########
site/streaming/index.html:
##########
@@ -125,135 +125,98 @@
<div class="row mt-4">
<div class="col-12 col-md-9">
<div class="jumbotron">
- <b>Spark streaming</b> makes it easy to build scalable fault-tolerant streaming
- applications.
+ <a href="/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.
</div>
<div class="row row-padded">
<div class="col-md-7 col-sm-7">
- <h2>Ease of use</h2>
- <p class="lead">
- Build applications through high-level operators.
- </p>
+ <h2>Ease to use</h2>
<p>
- Spark Streaming brings Apache Spark's
- <a href="/docs/latest/streaming-programming-guide.html">language-integrated API</a>
- to stream processing, letting you write streaming jobs the same way you write batch jobs.
- It supports Java, Scala and Python.
+ Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks
+ so that you can build streaming applications and pipelines without learning any new concepts or tools.
</p>
</div>
<div class="col-md-5 col-sm-5 col-padded-top col-center">
<div style="margin-top: 15px; text-align: left; display: inline-block;">
<div class="code">
- TwitterUtils.createStream(...)<br />
- .<span class="sparkop">filter</span>(<span class="closure">_.getText.contains("Spark")</span>)<br />
- .<span class="sparkop">countByWindow</span>(Seconds(5))
+ spark<br />
+ .<span class="sparkop">readStream</span><br />
+ .<span class="sparkop">select</span>(<span class="closure">cast("string").alias("jsonData")</span>)<br />
Review Comment:
> BTW, shall we put the <span class="closure"> around the double quotes? It will look more like the common highlight style
I missed this; I'll reflect this sooner than later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] gengliangwang commented on pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #389:
URL: https://github.com/apache/spark-website/pull/389#issuecomment-1130980503
Build passed. Merged to ASF site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on a diff in pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #389:
URL: https://github.com/apache/spark-website/pull/389#discussion_r876501507
##########
site/streaming/index.html:
##########
@@ -125,135 +125,98 @@
<div class="row mt-4">
<div class="col-12 col-md-9">
<div class="jumbotron">
- <b>Spark streaming</b> makes it easy to build scalable fault-tolerant streaming
- applications.
+ <a href="/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.
</div>
<div class="row row-padded">
<div class="col-md-7 col-sm-7">
- <h2>Ease of use</h2>
- <p class="lead">
- Build applications through high-level operators.
- </p>
+ <h2>Ease to use</h2>
<p>
- Spark Streaming brings Apache Spark's
- <a href="/docs/latest/streaming-programming-guide.html">language-integrated API</a>
- to stream processing, letting you write streaming jobs the same way you write batch jobs.
- It supports Java, Scala and Python.
+ Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks
+ so that you can build streaming applications and pipelines without learning any new concepts or tools.
</p>
</div>
<div class="col-md-5 col-sm-5 col-padded-top col-center">
<div style="margin-top: 15px; text-align: left; display: inline-block;">
<div class="code">
- TwitterUtils.createStream(...)<br />
- .<span class="sparkop">filter</span>(<span class="closure">_.getText.contains("Spark")</span>)<br />
- .<span class="sparkop">countByWindow</span>(Seconds(5))
+ spark<br />
+ .<span class="sparkop">readStream</span><br />
+ .<span class="sparkop">select</span>(<span class="closure">cast("string").alias("jsonData")</span>)<br />
Review Comment:
Just addressed this. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] gengliangwang merged pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
gengliangwang merged PR #389:
URL: https://github.com/apache/spark-website/pull/389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on a diff in pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #389:
URL: https://github.com/apache/spark-website/pull/389#discussion_r875607350
##########
site/streaming/index.html:
##########
@@ -125,135 +125,98 @@
<div class="row mt-4">
<div class="col-12 col-md-9">
<div class="jumbotron">
- <b>Spark streaming</b> makes it easy to build scalable fault-tolerant streaming
- applications.
+ <a href="/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.
</div>
<div class="row row-padded">
<div class="col-md-7 col-sm-7">
- <h2>Ease of use</h2>
- <p class="lead">
- Build applications through high-level operators.
- </p>
+ <h2>Ease to use</h2>
<p>
- Spark Streaming brings Apache Spark's
- <a href="/docs/latest/streaming-programming-guide.html">language-integrated API</a>
- to stream processing, letting you write streaming jobs the same way you write batch jobs.
- It supports Java, Scala and Python.
+ Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks
+ so that you can build streaming applications and pipelines without learning any new concepts or tools.
</p>
</div>
<div class="col-md-5 col-sm-5 col-padded-top col-center">
<div style="margin-top: 15px; text-align: left; display: inline-block;">
<div class="code">
- TwitterUtils.createStream(...)<br />
- .<span class="sparkop">filter</span>(<span class="closure">_.getText.contains("Spark")</span>)<br />
- .<span class="sparkop">countByWindow</span>(Seconds(5))
+ spark<br />
+ .<span class="sparkop">readStream</span><br />
+ .<span class="sparkop">select</span>(<span class="closure">cast("string").alias("jsonData")</span>)<br />
Review Comment:
Just fixed it; I used the simple batch query below for the verification:
```
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
val jsonSchema = new StructType().add("hello", StringType).add("year", IntegerType)
val df = spark.range(1)
.select(lit("{'hello': 'world', 'year': 2022}").alias("value")).select($"value".cast("string").alias("jsonData"))
.select(from_json($"jsonData", jsonSchema).alias("payload"))
df.show()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] dongjoon-hyun commented on pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #389:
URL: https://github.com/apache/spark-website/pull/389#issuecomment-1130218748
Thank you, @HeartSaVioR .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HeartSaVioR commented on a diff in pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on code in PR #389:
URL: https://github.com/apache/spark-website/pull/389#discussion_r875574247
##########
site/streaming/index.html:
##########
@@ -125,135 +125,98 @@
<div class="row mt-4">
<div class="col-12 col-md-9">
<div class="jumbotron">
- <b>Spark streaming</b> makes it easy to build scalable fault-tolerant streaming
- applications.
+ <a href="/docs/latest/structured-streaming-programming-guide.html">Spark Structured Streaming</a> makes it easy to build streaming applications and pipelines with the same and familiar Spark APIs.
</div>
<div class="row row-padded">
<div class="col-md-7 col-sm-7">
- <h2>Ease of use</h2>
- <p class="lead">
- Build applications through high-level operators.
- </p>
+ <h2>Ease to use</h2>
<p>
- Spark Streaming brings Apache Spark's
- <a href="/docs/latest/streaming-programming-guide.html">language-integrated API</a>
- to stream processing, letting you write streaming jobs the same way you write batch jobs.
- It supports Java, Scala and Python.
+ Spark Structured Streaming abstracts away complex streaming concepts such as incremental processing, checkpointing, and watermarks
+ so that you can build streaming applications and pipelines without learning any new concepts or tools.
</p>
</div>
<div class="col-md-5 col-sm-5 col-padded-top col-center">
<div style="margin-top: 15px; text-align: left; display: inline-block;">
<div class="code">
- TwitterUtils.createStream(...)<br />
- .<span class="sparkop">filter</span>(<span class="closure">_.getText.contains("Spark")</span>)<br />
- .<span class="sparkop">countByWindow</span>(Seconds(5))
+ spark<br />
+ .<span class="sparkop">readStream</span><br />
+ .<span class="sparkop">select</span>(<span class="closure">cast("string").alias("jsonData")</span>)<br />
Review Comment:
Nice finding! There's a missing piece - `col("value")` before cast. I'll reflect. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] gengliangwang commented on pull request #389: [SPARK-39219] Promote Structured Streaming over Spark Streaming (DStream)
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on PR #389:
URL: https://github.com/apache/spark-website/pull/389#issuecomment-1129679684
This is great! LGTM exception comments in the code example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org