You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2019/08/04 16:47:41 UTC
[spark] branch branch-2.4 updated: [SPARK-28609][DOC] Fix broken
styles/links and make up-to-date
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 04d9f8f [SPARK-28609][DOC] Fix broken styles/links and make up-to-date
04d9f8f is described below
commit 04d9f8f3cd565cbba13e47d20301ec5a0d98e75b
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sun Aug 4 09:42:47 2019 -0700
[SPARK-28609][DOC] Fix broken styles/links and make up-to-date
This PR aims to fix the broken styles/links and make the doc up-to-date for Apache Spark 2.4.4 and 3.0.0 release.
- `building-spark.md`
- `configuration.md`
- `sql-pyspark-pandas-with-arrow.md`
- `streaming-programming-guide.md`
- `structured-streaming-programming-guide.md` (1/2)
- `structured-streaming-programming-guide.md` (2/2)
- `submitting-applications.md`
Manual. Build the doc.
```
SKIP_API=1 jekyll build
```
Closes #25345 from dongjoon-hyun/SPARK-28609.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 4856c0e33abd1666f606d57102c198adf5cb5fc2)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/building-spark.md | 4 +-
docs/configuration.md | 2 +-
docs/sql-pyspark-pandas-with-arrow.md | 6 +--
docs/streaming-programming-guide.md | 2 +-
docs/structured-streaming-programming-guide.md | 64 ++++++++++++--------------
docs/submitting-applications.md | 2 +-
6 files changed, 37 insertions(+), 43 deletions(-)
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 7b9697c..8fa2cda 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -236,8 +236,8 @@ The run-tests script also can be limited to a specific Python version or a speci
To run the SparkR tests you will need to install the [knitr](https://cran.r-project.org/package=knitr), [rmarkdown](https://cran.r-project.org/package=rmarkdown), [testthat](https://cran.r-project.org/package=testthat), [e1071](https://cran.r-project.org/package=e1071) and [survival](https://cran.r-project.org/package=survival) packages first:
- R -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'e1071', 'survival'), repos='http://cran.us.r-project.org')"
- R -e "devtools::install_version('testthat', version = '1.0.2', repos='http://cran.us.r-project.org')"
+ Rscript -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'e1071', 'survival'), repos='https://cloud.r-project.org/')"
+ Rscript -e "devtools::install_version('testthat', version = '1.0.2', repos='https://cloud.r-project.org/')"
You can run just the SparkR tests using the command:
diff --git a/docs/configuration.md b/docs/configuration.md
index d481986..4db0ea6 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -530,7 +530,7 @@ Apart from these, the following properties are also available, and may be useful
option <code>--repositories</code> or <code>spark.jars.repositories</code> will also be included.
Useful for allowing Spark to resolve artifacts from behind a firewall e.g. via an in-house
artifact server like Artifactory. Details on the settings file format can be
- found at http://ant.apache.org/ivy/history/latest-milestone/settings.html
+ found at <a href="http://ant.apache.org/ivy/history/latest-milestone/settings.html">Settings Files</a>
</td>
</tr>
<tr>
diff --git a/docs/sql-pyspark-pandas-with-arrow.md b/docs/sql-pyspark-pandas-with-arrow.md
index d04b955..adfd2bd 100644
--- a/docs/sql-pyspark-pandas-with-arrow.md
+++ b/docs/sql-pyspark-pandas-with-arrow.md
@@ -29,11 +29,11 @@ You can install using pip or conda from the conda-forge channel. See PyArrow
Arrow is available as an optimization when converting a Spark DataFrame to a Pandas DataFrame
using the call `toPandas()` and when creating a Spark DataFrame from a Pandas DataFrame with
`createDataFrame(pandas_df)`. To use Arrow when executing these calls, users need to first set
-the Spark configuration 'spark.sql.execution.arrow.enabled' to 'true'. This is disabled by default.
+the Spark configuration `spark.sql.execution.arrow.enabled` to `true`. This is disabled by default.
-In addition, optimizations enabled by 'spark.sql.execution.arrow.enabled' could fallback automatically
+In addition, optimizations enabled by `spark.sql.execution.arrow.enabled` could fallback automatically
to non-Arrow optimization implementation if an error occurs before the actual computation within Spark.
-This can be controlled by 'spark.sql.execution.arrow.fallback.enabled'.
+This can be controlled by `spark.sql.execution.arrow.fallback.enabled`.
<div class="codetabs">
<div data-lang="python" markdown="1">
diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md
index 87ff4be..6e6ae03 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -432,7 +432,7 @@ val ssc = new StreamingContext(conf, Seconds(1))
{% endhighlight %}
The `appName` parameter is a name for your application to show on the cluster UI.
-`master` is a [Spark, Mesos or YARN cluster URL](submitting-applications.html#master-urls),
+`master` is a [Spark, Mesos, Kubernetes or YARN cluster URL](submitting-applications.html#master-urls),
or a special __"local[\*]"__ string to run in local mode. In practice, when running on a cluster,
you will not want to hardcode `master` in the program,
but rather [launch the application with `spark-submit`](submitting-applications.html) and
diff --git a/docs/structured-streaming-programming-guide.md b/docs/structured-streaming-programming-guide.md
index a93f65b..2c3348a 100644
--- a/docs/structured-streaming-programming-guide.md
+++ b/docs/structured-streaming-programming-guide.md
@@ -12,7 +12,7 @@ Structured Streaming is a scalable and fault-tolerant stream processing engine b
Internally, by default, Structured Streaming queries are processed using a *micro-batch processing* engine, which processes data streams as a series of small batch jobs thereby achieving end-to-end latencies as low as 100 milliseconds and exactly-once fault-tolerance guarantees. However, since Spark 2.3, we have introduced a new low-latency processing mode called **Continuous Processing**, which can achieve end-to-end latencies as low as 1 millisecond with at-least-once guarantees. Witho [...]
-In this guide, we are going to walk you through the programming model and the APIs. We are going to explain the concepts mostly using the default micro-batch processing model, and then [later](#continuous-processing-experimental) discuss Continuous Processing model. First, let's start with a simple example of a Structured Streaming query - a streaming word count.
+In this guide, we are going to walk you through the programming model and the APIs. We are going to explain the concepts mostly using the default micro-batch processing model, and then [later](#continuous-processing) discuss Continuous Processing model. First, let's start with a simple example of a Structured Streaming query - a streaming word count.
# Quick Example
Let’s say you want to maintain a running word count of text data received from a data server listening on a TCP socket. Let’s see how you can express this using Structured Streaming. You can see the full code in
@@ -1480,9 +1480,9 @@ Additional details on supported joins:
- Joins can be cascaded, that is, you can do `df1.join(df2, ...).join(df3, ...).join(df4, ....)`.
-- As of Spark 2.3, you can use joins only when the query is in Append output mode. Other output modes are not yet supported.
+- As of Spark 2.4, you can use joins only when the query is in Append output mode. Other output modes are not yet supported.
-- As of Spark 2.3, you cannot use other non-map-like operations before joins. Here are a few examples of
+- As of Spark 2.4, you cannot use other non-map-like operations before joins. Here are a few examples of
what cannot be used.
- Cannot use streaming aggregations before joins.
@@ -2174,34 +2174,34 @@ The function offers a simple way to express your processing logic but does not a
deduplicate generated data when failures cause reprocessing of some input data.
For that situation you must specify the processing logic in an object.
-1. The function takes a row as input.
+- First, the function takes a row as input.
- {% highlight python %}
- def process_row(row):
- # Write row to storage
- pass
-
- query = streamingDF.writeStream.foreach(process_row).start()
- {% endhighlight %}
+{% highlight python %}
+def process_row(row):
+ # Write row to storage
+ pass
-2. The object has a process method and optional open and close methods:
+query = streamingDF.writeStream.foreach(process_row).start()
+{% endhighlight %}
- {% highlight python %}
- class ForeachWriter:
- def open(self, partition_id, epoch_id):
- # Open connection. This method is optional in Python.
- pass
-
- def process(self, row):
- # Write row to connection. This method is NOT optional in Python.
- pass
-
- def close(self, error):
- # Close the connection. This method in optional in Python.
- pass
+- Second, the object has a process method and optional open and close methods:
+
+{% highlight python %}
+class ForeachWriter:
+ def open(self, partition_id, epoch_id):
+ # Open connection. This method is optional in Python.
+ pass
+
+ def process(self, row):
+ # Write row to connection. This method is NOT optional in Python.
+ pass
+
+ def close(self, error):
+ # Close the connection. This method in optional in Python.
+ pass
- query = streamingDF.writeStream.foreach(ForeachWriter()).start()
- {% endhighlight %}
+query = streamingDF.writeStream.foreach(ForeachWriter()).start()
+{% endhighlight %}
</div>
<div data-lang="r" markdown="1">
@@ -2291,7 +2291,7 @@ Here are the different kinds of triggers that are supported.
<td><b>Continuous with fixed checkpoint interval</b><br/><i>(experimental)</i></td>
<td>
The query will be executed in the new low-latency, continuous processing mode. Read more
- about this in the <a href="#continuous-processing-experimental">Continuous Processing section</a> below.
+ about this in the <a href="#continuous-processing">Continuous Processing section</a> below.
</td>
</tr>
</table>
@@ -3033,12 +3033,6 @@ import org.apache.spark.sql.streaming.Trigger
spark
.readStream
- .format("rate")
- .option("rowsPerSecond", "10")
- .option("")
-
-spark
- .readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host1:port1,host2:port2")
.option("subscribe", "topic1")
@@ -3096,7 +3090,7 @@ A checkpoint interval of 1 second means that the continuous processing engine wi
## Supported Queries
{:.no_toc}
-As of Spark 2.3, only the following type of queries are supported in the continuous processing mode.
+As of Spark 2.4, only the following type of queries are supported in the continuous processing mode.
- *Operations*: Only map-like Dataset/DataFrame operations are supported in continuous mode, that is, only projections (`select`, `map`, `flatMap`, `mapPartitions`, etc.) and selections (`where`, `filter`, etc.).
+ All SQL functions are supported except aggregation functions (since aggregations are not yet supported), `current_timestamp()` and `current_date()` (deterministic computations using time is challenging).
diff --git a/docs/submitting-applications.md b/docs/submitting-applications.md
index 77aa083..34cea82 100644
--- a/docs/submitting-applications.md
+++ b/docs/submitting-applications.md
@@ -167,7 +167,7 @@ The master URL passed to Spark can be in one of the following formats:
</td></tr>
<tr><td> <code>k8s://HOST:PORT</code> </td><td> Connect to a <a href="running-on-kubernetes.html">Kubernetes</a> cluster in
<code>cluster</code> mode. Client mode is currently unsupported and will be supported in future releases.
- The <code>HOST</code> and <code>PORT</code> refer to the [Kubernetes API Server](https://kubernetes.io/docs/reference/generated/kube-apiserver/).
+ The <code>HOST</code> and <code>PORT</code> refer to the <a href="https://kubernetes.io/docs/reference/generated/kube-apiserver/">Kubernetes API Server</a>.
It connects using TLS by default. In order to force it to use an unsecured connection, you can use
<code>k8s://http://HOST:PORT</code>.
</td></tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org