You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by zs...@apache.org on 2017/06/20 20:07:54 UTC
spark git commit: [SPARK-21123][DOCS][STRUCTURED STREAMING] Options
for file stream source are in a wrong table - version to fix 2.1
Repository: spark
Updated Branches:
refs/heads/branch-2.1 7799f35dd -> 8923bac1e
[SPARK-21123][DOCS][STRUCTURED STREAMING] Options for file stream source are in a wrong table - version to fix 2.1
## What changes were proposed in this pull request?
The description for several options of File Source for structured streaming appeared in the File Sink description instead.
This commit continues on PR #18342 and targets the fixes for the documentation of version spark version 2.1
## How was this patch tested?
Built the documentation by SKIP_API=1 jekyll build and visually inspected the structured streaming programming guide.
zsxwing This is the PR to fix version 2.1 as discussed in PR #18342
Author: assafmendelson <as...@gmail.com>
Closes #18363 from assafmendelson/spark-21123-for-spark2.1.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8923bac1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8923bac1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8923bac1
Branch: refs/heads/branch-2.1
Commit: 8923bac1e895e57ce2d9ef6aea31e13e390be6da
Parents: 7799f35
Author: assafmendelson <as...@gmail.com>
Authored: Tue Jun 20 13:07:51 2017 -0700
Committer: Shixiong Zhu <sh...@databricks.com>
Committed: Tue Jun 20 13:07:51 2017 -0700
----------------------------------------------------------------------
docs/structured-streaming-programming-guide.md | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/8923bac1/docs/structured-streaming-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/structured-streaming-programming-guide.md b/docs/structured-streaming-programming-guide.md
index da5c234..e536f5d 100644
--- a/docs/structured-streaming-programming-guide.md
+++ b/docs/structured-streaming-programming-guide.md
@@ -449,6 +449,10 @@ Here are the details of all the sources in Spark.
<td><b>File source</b></td>
<td>
<code>path</code>: path to the input directory, and common to all file formats.
+ <br/>
+ <code>maxFilesPerTrigger</code>: maximum number of new files to be considered in every trigger (default: no max)
+ <br/>
+ <code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files(default: false)
<br/><br/>
For file-format-specific options, see the related methods in <code>DataStreamReader</code>
(<a href="api/scala/index.html#org.apache.spark.sql.streaming.DataStreamReader">Scala</a>/<a href="api/java/org/apache/spark/sql/streaming/DataStreamReader.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.streaming.DataStreamReader">Python</a>).
@@ -1076,9 +1080,6 @@ Here are the details of all the sinks in Spark.
<td>Append</td>
<td>
<code>path</code>: path to the output directory, must be specified.
- <code>maxFilesPerTrigger</code>: maximum number of new files to be considered in every trigger (default: no max)
- <br/>
- <code>latestFirst</code>: whether to processs the latest new files first, useful when there is a large backlog of files(default: false)
<br/><br/>
For file-format-specific options, see the related methods in DataFrameWriter
(<a href="api/scala/index.html#org.apache.spark.sql.DataFrameWriter">Scala</a>/<a href="api/java/org/apache/spark/sql/DataFrameWriter.html">Java</a>/<a href="api/python/pyspark.sql.html#pyspark.sql.DataFrameWriter">Python</a>).
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org