You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/08/23 20:21:48 UTC
spark git commit: [MINOR][DOC] Use standard quotes instead of "curly
quote" marks from Mac in structured streaming programming guides
Repository: spark
Updated Branches:
refs/heads/master 8fd63e808 -> 588559911
[MINOR][DOC] Use standard quotes instead of "curly quote" marks from Mac in structured streaming programming guides
## What changes were proposed in this pull request?
This PR fixes curly quotes (`\u201c` and `\u201d` ) to standard quotes (`"`).
This will be a actual problem when users copy and paste the examples. This would not work.
This seems only happening in `structured-streaming-programming-guide.md`.
## How was this patch tested?
Manually built.
This will change some examples to be correctly marked down as below:
to
Author: hyukjinkwon <gu...@gmail.com>
Closes #14770 from HyukjinKwon/minor-quotes.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/58855991
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/58855991
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/58855991
Branch: refs/heads/master
Commit: 588559911de94bbe0932526ee1e1dd36a581a423
Parents: 8fd63e8
Author: hyukjinkwon <gu...@gmail.com>
Authored: Tue Aug 23 21:21:43 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Tue Aug 23 21:21:43 2016 +0100
----------------------------------------------------------------------
docs/structured-streaming-programming-guide.md | 38 ++++++++++-----------
1 file changed, 19 insertions(+), 19 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/58855991/docs/structured-streaming-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/structured-streaming-programming-guide.md b/docs/structured-streaming-programming-guide.md
index 226ff74..090b14f 100644
--- a/docs/structured-streaming-programming-guide.md
+++ b/docs/structured-streaming-programming-guide.md
@@ -88,7 +88,7 @@ val words = lines.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()
{% endhighlight %}
-This `lines` DataFrame represents an unbounded table containing the streaming text data. This table contains one column of strings named \u201cvalue\u201d, and each line in the streaming text data becomes a row in the table. Note, that this is not currently receiving any data as we are just setting up the transformation, and have not yet started it. Next, we have converted the DataFrame to a Dataset of String using `.as[String]`, so that we can apply the `flatMap` operation to split each line into multiple words. The resultant `words` Dataset contains all the words. Finally, we have defined the `wordCounts` DataFrame by grouping by the unique values in the Dataset and counting them. Note that this is a streaming DataFrame which represents the running word counts of the stream.
+This `lines` DataFrame represents an unbounded table containing the streaming text data. This table contains one column of strings named "value", and each line in the streaming text data becomes a row in the table. Note, that this is not currently receiving any data as we are just setting up the transformation, and have not yet started it. Next, we have converted the DataFrame to a Dataset of String using `.as[String]`, so that we can apply the `flatMap` operation to split each line into multiple words. The resultant `words` Dataset contains all the words. Finally, we have defined the `wordCounts` DataFrame by grouping by the unique values in the Dataset and counting them. Note that this is a streaming DataFrame which represents the running word counts of the stream.
</div>
<div data-lang="java" markdown="1">
@@ -117,7 +117,7 @@ Dataset<String> words = lines
Dataset<Row> wordCounts = words.groupBy("value").count();
{% endhighlight %}
-This `lines` DataFrame represents an unbounded table containing the streaming text data. This table contains one column of strings named \u201cvalue\u201d, and each line in the streaming text data becomes a row in the table. Note, that this is not currently receiving any data as we are just setting up the transformation, and have not yet started it. Next, we have converted the DataFrame to a Dataset of String using `.as(Encoders.STRING())`, so that we can apply the `flatMap` operation to split each line into multiple words. The resultant `words` Dataset contains all the words. Finally, we have defined the `wordCounts` DataFrame by grouping by the unique values in the Dataset and counting them. Note that this is a streaming DataFrame which represents the running word counts of the stream.
+This `lines` DataFrame represents an unbounded table containing the streaming text data. This table contains one column of strings named "value", and each line in the streaming text data becomes a row in the table. Note, that this is not currently receiving any data as we are just setting up the transformation, and have not yet started it. Next, we have converted the DataFrame to a Dataset of String using `.as(Encoders.STRING())`, so that we can apply the `flatMap` operation to split each line into multiple words. The resultant `words` Dataset contains all the words. Finally, we have defined the `wordCounts` DataFrame by grouping by the unique values in the Dataset and counting them. Note that this is a streaming DataFrame which represents the running word counts of the stream.
</div>
<div data-lang="python" markdown="1">
@@ -142,12 +142,12 @@ words = lines.select(
wordCounts = words.groupBy('word').count()
{% endhighlight %}
-This `lines` DataFrame represents an unbounded table containing the streaming text data. This table contains one column of strings named \u201cvalue\u201d, and each line in the streaming text data becomes a row in the table. Note, that this is not currently receiving any data as we are just setting up the transformation, and have not yet started it. Next, we have used two built-in SQL functions - split and explode, to split each line into multiple rows with a word each. In addition, we use the function `alias` to name the new column as \u201cword\u201d. Finally, we have defined the `wordCounts` DataFrame by grouping by the unique values in the Dataset and counting them. Note that this is a streaming DataFrame which represents the running word counts of the stream.
+This `lines` DataFrame represents an unbounded table containing the streaming text data. This table contains one column of strings named "value", and each line in the streaming text data becomes a row in the table. Note, that this is not currently receiving any data as we are just setting up the transformation, and have not yet started it. Next, we have used two built-in SQL functions - split and explode, to split each line into multiple rows with a word each. In addition, we use the function `alias` to name the new column as "word". Finally, we have defined the `wordCounts` DataFrame by grouping by the unique values in the Dataset and counting them. Note that this is a streaming DataFrame which represents the running word counts of the stream.
</div>
</div>
-We have now set up the query on the streaming data. All that is left is to actually start receiving data and computing the counts. To do this, we set it up to print the complete set of counts (specified by `outputMode(\u201ccomplete\u201d)`) to the console every time they are updated. And then start the streaming computation using `start()`.
+We have now set up the query on the streaming data. All that is left is to actually start receiving data and computing the counts. To do this, we set it up to print the complete set of counts (specified by `outputMode("complete")`) to the console every time they are updated. And then start the streaming computation using `start()`.
<div class="codetabs">
<div data-lang="scala" markdown="1">
@@ -361,16 +361,16 @@ table, and Spark runs it as an *incremental* query on the *unbounded* input
table. Let\u2019s understand this model in more detail.
## Basic Concepts
-Consider the input data stream as the \u201cInput Table\u201d. Every data item that is
+Consider the input data stream as the "Input Table". Every data item that is
arriving on the stream is like a new row being appended to the Input Table.
-A query on the input will generate the \u201cResult Table\u201d. Every trigger interval (say, every 1 second), new rows get appended to the Input Table, which eventually updates the Result Table. Whenever the result table gets updated, we would want to write the changed result rows to an external sink.
+A query on the input will generate the "Result Table". Every trigger interval (say, every 1 second), new rows get appended to the Input Table, which eventually updates the Result Table. Whenever the result table gets updated, we would want to write the changed result rows to an external sink.
-The \u201cOutput\u201d is defined as what gets written out to the external storage. The output can be defined in different modes
+The "Output" is defined as what gets written out to the external storage. The output can be defined in different modes
- *Complete Mode* - The entire updated Result Table will be written to the external storage. It is up to the storage connector to decide how to handle writing of the entire table.
@@ -386,7 +386,7 @@ the final `wordCounts` DataFrame is the result table. Note that the query on
streaming `lines` DataFrame to generate `wordCounts` is *exactly the same* as
it would be a static DataFrame. However, when this query is started, Spark
will continuously check for new data from the socket connection. If there is
-new data, Spark will run an \u201cincremental\u201d query that combines the previous
+new data, Spark will run an "incremental" query that combines the previous
running counts with the new data to compute updated counts, as shown below.
@@ -682,8 +682,8 @@ Streaming DataFrames can be joined with static DataFrames to create new streamin
val staticDf = spark.read. ...
val streamingDf = spark.readStream. ...
-streamingDf.join(staticDf, \u201ctype\u201d) // inner equi-join with a static DF
-streamingDf.join(staticDf, \u201ctype\u201d, \u201cright_join\u201d) // right outer join with a static DF
+streamingDf.join(staticDf, "type") // inner equi-join with a static DF
+streamingDf.join(staticDf, "type", "right_join") // right outer join with a static DF
{% endhighlight %}
@@ -789,7 +789,7 @@ Here is a table of all the sinks, and the corresponding settings.
<tr>
<td><b>File Sink</b><br/>(only parquet in Spark 2.0)</td>
<td>Append</td>
- <td><pre>writeStream<br/> .format(\u201cparquet\u201d)<br/> .start()</pre></td>
+ <td><pre>writeStream<br/> .format("parquet")<br/> .start()</pre></td>
<td>Yes</td>
<td>Supports writes to partitioned tables. Partitioning by time may be useful.</td>
</tr>
@@ -803,14 +803,14 @@ Here is a table of all the sinks, and the corresponding settings.
<tr>
<td><b>Console Sink</b></td>
<td>Append, Complete</td>
- <td><pre>writeStream<br/> .format(\u201cconsole\u201d)<br/> .start()</pre></td>
+ <td><pre>writeStream<br/> .format("console")<br/> .start()</pre></td>
<td>No</td>
<td></td>
</tr>
<tr>
<td><b>Memory Sink</b></td>
<td>Append, Complete</td>
- <td><pre>writeStream<br/> .format(\u201cmemory\u201d)<br/> .queryName(\u201ctable\u201d)<br/> .start()</pre></td>
+ <td><pre>writeStream<br/> .format("memory")<br/> .queryName("table")<br/> .start()</pre></td>
<td>No</td>
<td>Saves the output data as a table, for interactive querying. Table name is the query name.</td>
</tr>
@@ -839,7 +839,7 @@ noAggDF
.start()
// ========== DF with aggregation ==========
-val aggDF = df.groupBy(\u201cdevice\u201d).count()
+val aggDF = df.groupBy("device").count()
// Print updated aggregations to console
aggDF
@@ -879,7 +879,7 @@ noAggDF
.start();
// ========== DF with aggregation ==========
-Dataset<Row> aggDF = df.groupBy(\u201cdevice\u201d).count();
+Dataset<Row> aggDF = df.groupBy("device").count();
// Print updated aggregations to console
aggDF
@@ -919,7 +919,7 @@ noAggDF\
.start()
# ========== DF with aggregation ==========
-aggDF = df.groupBy(\u201cdevice\u201d).count()
+aggDF = df.groupBy("device").count()
# Print updated aggregations to console
aggDF\
@@ -1095,7 +1095,7 @@ In case of a failure or intentional shutdown, you can recover the previous progr
aggDF
.writeStream
.outputMode("complete")
- .option(\u201ccheckpointLocation\u201d, \u201cpath/to/HDFS/dir\u201d)
+ .option("checkpointLocation", "path/to/HDFS/dir")
.format("memory")
.start()
{% endhighlight %}
@@ -1107,7 +1107,7 @@ aggDF
aggDF
.writeStream()
.outputMode("complete")
- .option(\u201ccheckpointLocation\u201d, \u201cpath/to/HDFS/dir\u201d)
+ .option("checkpointLocation", "path/to/HDFS/dir")
.format("memory")
.start();
{% endhighlight %}
@@ -1119,7 +1119,7 @@ aggDF
aggDF\
.writeStream()\
.outputMode("complete")\
- .option(\u201ccheckpointLocation\u201d, \u201cpath/to/HDFS/dir\u201d)\
+ .option("checkpointLocation", "path/to/HDFS/dir")\
.format("memory")\
.start()
{% endhighlight %}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org