You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2018/01/17 02:18:28 UTC
[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/20286
[SPARK-23119][SS] Minor fixes to V2 streaming APIs
## What changes were proposed in this pull request?
- Added `@InterfaceStability.Evolving` annotations
- Improved docs.
## How was this patch tested?
Existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-23119
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20286.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20286
----
commit f6dfa58e2bc47f447c02d54a834e164e70083db9
Author: Tathagata Das <ta...@...>
Date: 2018-01-17T01:59:25Z
Minor fixes to V2 APIs
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20286
jenkins retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20286
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20286
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20286
**[Test build #86249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86249/testReport)** for PR 20286 at commit [`f6dfa58`](https://github.com/apache/spark/commit/f6dfa58e2bc47f447c02d54a834e164e70083db9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20286#discussion_r161976455
--- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/reader/Offset.java ---
@@ -17,12 +17,20 @@
package org.apache.spark.sql.sources.v2.streaming.reader;
+import org.apache.spark.annotation.InterfaceStability;
+
/**
- * An abstract representation of progress through a [[MicroBatchReader]] or [[ContinuousReader]].
- * During execution, Offsets provided by the data source implementation will be logged and used as
- * restart checkpoints. Sources should provide an Offset implementation which they can use to
- * reconstruct the stream position where the offset was taken.
+ * An abstract representation of progress through a {@link MicroBatchReader} or
+ * {@link ContinuousReader}.
+ * During execution, offsets provided by the data source implementation will be logged and used as
+ * restart checkpoints. Each source should provide an offset implementation which the source can use
+ * to reconstruct a position in the stream up to which data has been seen/processed.
+ *
+ * Note: This class currently extends {@link org.apache.spark.sql.execution.streaming.Offset} to
+ * maintain compatibility with DataSource V1 APIs. This will be extension will be removed once we
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20286
**[Test build #86224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86224/testReport)** for PR 20286 at commit [`f6dfa58`](https://github.com/apache/spark/commit/f6dfa58e2bc47f447c02d54a834e164e70083db9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by smurakozi <gi...@git.apache.org>.
Github user smurakozi commented on the issue:
https://github.com/apache/spark/pull/20286
I think this change is OK, except the nits zsxwing already noted.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20286
**[Test build #86292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86292/testReport)** for PR 20286 at commit [`3b3029b`](https://github.com/apache/spark/commit/3b3029bdaf6f36e02cfc3c493c42e39cc86c3057).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20286
**[Test build #86224 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86224/testReport)** for PR 20286 at commit [`f6dfa58`](https://github.com/apache/spark/commit/f6dfa58e2bc47f447c02d54a834e164e70083db9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20286
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20286
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20286
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86292/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20286
LGTM. Merging to master and 2.3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20286
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20286#discussion_r161976371
--- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/reader/ContinuousReader.java ---
@@ -27,11 +28,15 @@
* interface to allow reading in a continuous processing mode stream.
*
* Implementations must ensure each read task output is a {@link ContinuousDataReader}.
+ *
+ * Note: This class currently extends {@link BaseStreamingSource} to maintain compatibility with
+ * DataSource V1 APIs. This will be extension will be removed once we get rid of V1 completely.
--- End diff --
nit: This ~~will be~~ extension
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20286
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86224/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20286
@jose-torres @zsxwing Can you take a look.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20286
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86249/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20286: [SPARK-23119][SS] Minor fixes to V2 streaming API...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/20286#discussion_r161976313
--- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/streaming/reader/MicroBatchReader.java ---
@@ -25,7 +26,11 @@
/**
* A mix-in interface for {@link DataSourceV2Reader}. Data source readers can implement this
* interface to indicate they allow micro-batch streaming reads.
+ *
+ * Note: This class currently extends {@link BaseStreamingSource} to maintain compatibility with
+ * DataSource V1 APIs. This will be extension will be removed once we get rid of V1 completely.
--- End diff --
ditto
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20286
**[Test build #86292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86292/testReport)** for PR 20286 at commit [`3b3029b`](https://github.com/apache/spark/commit/3b3029bdaf6f36e02cfc3c493c42e39cc86c3057).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20286: [SPARK-23119][SS] Minor fixes to V2 streaming APIs
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20286
**[Test build #86249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86249/testReport)** for PR 20286 at commit [`f6dfa58`](https://github.com/apache/spark/commit/f6dfa58e2bc47f447c02d54a834e164e70083db9).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org