You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2014/10/08 02:58:12 UTC
[GitHub] spark pull request: WIP: Spark 3754 spark streaming file system ap...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/2703
WIP: Spark 3754 spark streaming file system api callable from java
This is a WIP of making the filesystem api callable for the Java version of Spark Streaming.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark SPARK-3754-spark-streaming-fileSystemAPI-callable-from-java
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2703.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2703
----
commit 2632a1e0c747d3734150c4ee3a6bc1c7537983e8
Author: Holden Karau <ho...@pigscanfly.ca>
Date: 2014-10-08T00:39:55Z
Draft of adding filestream support to Java API with tests
commit 95f2bfee299792810379d28da1aa8f6eccb5c03e
Author: Holden Karau <ho...@pigscanfly.ca>
Date: 2014-10-08T00:55:42Z
hmm
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-60327025
Thanks @holdenk for doing this. This is a great fix.
But since this is an API change which breaks binary compatibility (I am okay with it, as no one is probably using this method), you have to add exclusions in the mima.
Run the mima test with `dev/mima`, see the error and use that output to add to the exclusions in `project/MimaExcludes`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r22290108
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala ---
@@ -250,19 +250,19 @@ class JavaStreamingContext(val ssc: StreamingContext) extends Closeable {
* Files must be written to the monitored directory by "moving" them from another
* location within the same file system. File names starting with . are ignored.
* @param directory HDFS directory to monitor for new file
- * @tparam K Key type for reading HDFS file
- * @tparam V Value type for reading HDFS file
- * @tparam F Input format for reading HDFS file
+ * @param inputFormatClass Input format for reading HDFS file
+ * @param keyClass Key type for reading HDFS file
+ * @param valueClass Value type for reading HDFS file
*/
def fileStream[K, V, F <: NewInputFormat[K, V]](
- directory: String): JavaPairInputDStream[K, V] = {
- implicit val cmk: ClassTag[K] =
- implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[K]]
- implicit val cmv: ClassTag[V] =
- implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[V]]
- implicit val cmf: ClassTag[F] =
- implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[F]]
- ssc.fileStream[K, V, F](directory)
+ directory: String,
+ inputFormatClass: Class[F],
+ keyClass: Class[K],
+ valueClass: Class[V], newFilesOnly: Boolean = true): JavaPairInputDStream[K, V] = {
--- End diff --
Correction on this comment. newFilesOnly should be exposed as it is exposed in the Scala api.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-68417363
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-61902898
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22958/consoleFull) for PR 2703 at commit [`3a2cd21`](https://github.com/apache/spark/commit/3a2cd21ec6672ecfb780d3a98fc5c1b5907c12e2).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r20126383
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -331,8 +331,8 @@ class StreamingContext private[streaming] (
K: ClassTag,
V: ClassTag,
F <: NewInputFormat[K, V]: ClassTag
- ] (directory: String): InputDStream[(K, V)] = {
- new FileInputDStream[K, V, F](this, directory)
+ ] (directory: String, newFilesOnly: Boolean = true): InputDStream[(K, V)] = {
--- End diff --
Hey @holdenk, although the underlying fileinputdstream supported it, it was not exposed in the API. I am not sure its worth exposing this kind of functionality. So lets not change this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r19435702
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -331,8 +331,8 @@ class StreamingContext private[streaming] (
K: ClassTag,
V: ClassTag,
F <: NewInputFormat[K, V]: ClassTag
- ] (directory: String): InputDStream[(K, V)] = {
- new FileInputDStream[K, V, F](this, directory)
+ ] (directory: String, newFilesOnly: Boolean = true): InputDStream[(K, V)] = {
--- End diff --
the use case of exposing this was so that I could expose equivelent java functionality to what was available in Scala
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-68415747
I'll take a look this weekend when I have some time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: WIP: Spark 3754 spark streaming file system ap...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58446887
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21496/consoleFull) for PR 2703 at commit [`47cfdd5`](https://github.com/apache/spark/commit/47cfdd58075445822579c9271ef12521a7565ab0).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r19316615
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -331,8 +331,8 @@ class StreamingContext private[streaming] (
K: ClassTag,
V: ClassTag,
F <: NewInputFormat[K, V]: ClassTag
- ] (directory: String): InputDStream[(K, V)] = {
- new FileInputDStream[K, V, F](this, directory)
+ ] (directory: String, newFilesOnly: Boolean = true): InputDStream[(K, V)] = {
--- End diff --
I am not sure whether there is any use case for exposing this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r19316316
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -331,8 +331,8 @@ class StreamingContext private[streaming] (
K: ClassTag,
V: ClassTag,
F <: NewInputFormat[K, V]: ClassTag
- ] (directory: String): InputDStream[(K, V)] = {
- new FileInputDStream[K, V, F](this, directory)
+ ] (directory: String, newFilesOnly: Boolean = true): InputDStream[(K, V)] = {
--- End diff --
This does not seem to be related to SPARK 3754
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-68415266
Ping?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r22290121
--- Diff: streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -1703,6 +1710,65 @@ public void testTextFileStream() {
JavaDStream<String> test = ssc.textFileStream("/tmp/foo");
}
+
+ @Test
+ public void testFileStream() throws Exception {
+ // Disable manual clock as FileInputDStream does not work with manual clock
+ System.setProperty("spark.streaming.clock", "org.apache.spark.streaming.util.SystemClock");
+ ssc = new JavaStreamingContext("local[2]", "test", new Duration(1000));
+ ssc.checkpoint("checkpoint");
+ // Set up some sequence files for streaming to read in
+ List<Tuple2<Long, Integer>> test_input = new ArrayList<Tuple2<Long, Integer> >();
+ test_input.add(new Tuple2(1L, 123456));
+ test_input.add(new Tuple2(2L, 123456));
+ JavaPairRDD<Long, Integer> rdd = ssc.sc().parallelizePairs(test_input);
+ File tempDir = Files.createTempDir();
+ JavaPairRDD<LongWritable, IntWritable> saveable = rdd.mapToPair(
+ new PairFunction<Tuple2<Long, Integer>, LongWritable, IntWritable>() {
+ public Tuple2<LongWritable, IntWritable> call(Tuple2<Long, Integer> record) {
+ return new Tuple2(new LongWritable(record._1), new IntWritable(record._2));
+ }});
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/1/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/2/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+
+ // Construct a file stream from the above saved data
+ JavaPairDStream<LongWritable, IntWritable> testRaw = ssc.fileStream(
+ tempDir.getAbsolutePath() + "/" , SequenceFileInputFormat.class, LongWritable.class,
+ IntWritable.class, false);
+ JavaPairDStream<Long, Integer> test = testRaw.mapToPair(
+ new PairFunction<Tuple2<LongWritable, IntWritable>, Long, Integer>() {
+ public Tuple2<Long, Integer> call(Tuple2<LongWritable, IntWritable> input) {
+ return new Tuple2(input._1().get(), input._2().get());
+ }
+ });
+ final Accumulator<Integer> elem = ssc.sc().intAccumulator(0);
--- End diff --
Why is it not possible to just call rdd.count() and add up the counts in a global counter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r20126407
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala ---
@@ -250,19 +250,19 @@ class JavaStreamingContext(val ssc: StreamingContext) extends Closeable {
* Files must be written to the monitored directory by "moving" them from another
* location within the same file system. File names starting with . are ignored.
* @param directory HDFS directory to monitor for new file
- * @tparam K Key type for reading HDFS file
- * @tparam V Value type for reading HDFS file
- * @tparam F Input format for reading HDFS file
+ * @param inputFormatClass Input format for reading HDFS file
+ * @param keyClass Key type for reading HDFS file
+ * @param valueClass Value type for reading HDFS file
*/
def fileStream[K, V, F <: NewInputFormat[K, V]](
- directory: String): JavaPairInputDStream[K, V] = {
- implicit val cmk: ClassTag[K] =
- implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[K]]
- implicit val cmv: ClassTag[V] =
- implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[V]]
- implicit val cmf: ClassTag[F] =
- implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[F]]
- ssc.fileStream[K, V, F](directory)
+ directory: String,
+ inputFormatClass: Class[F],
+ keyClass: Class[K],
+ valueClass: Class[V], newFilesOnly: Boolean = true): JavaPairInputDStream[K, V] = {
--- End diff --
Same reason as above, no need to expose this functionality of newFilesOnly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-62360202
[Test build #23146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23146/consoleFull) for PR 2703 at commit [`efac1fc`](https://github.com/apache/spark/commit/efac1fc5f2f46762ab339838b5c1a75707182e4a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-73783300
This has already been fixed by https://github.com/apache/spark/pull/4101 Can you please close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58451978
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21496/Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r22290126
--- Diff: streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -1703,6 +1710,65 @@ public void testTextFileStream() {
JavaDStream<String> test = ssc.textFileStream("/tmp/foo");
}
+
+ @Test
+ public void testFileStream() throws Exception {
+ // Disable manual clock as FileInputDStream does not work with manual clock
+ System.setProperty("spark.streaming.clock", "org.apache.spark.streaming.util.SystemClock");
+ ssc = new JavaStreamingContext("local[2]", "test", new Duration(1000));
+ ssc.checkpoint("checkpoint");
+ // Set up some sequence files for streaming to read in
+ List<Tuple2<Long, Integer>> test_input = new ArrayList<Tuple2<Long, Integer> >();
+ test_input.add(new Tuple2(1L, 123456));
+ test_input.add(new Tuple2(2L, 123456));
+ JavaPairRDD<Long, Integer> rdd = ssc.sc().parallelizePairs(test_input);
+ File tempDir = Files.createTempDir();
+ JavaPairRDD<LongWritable, IntWritable> saveable = rdd.mapToPair(
+ new PairFunction<Tuple2<Long, Integer>, LongWritable, IntWritable>() {
+ public Tuple2<LongWritable, IntWritable> call(Tuple2<Long, Integer> record) {
+ return new Tuple2(new LongWritable(record._1), new IntWritable(record._2));
+ }});
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/1/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/2/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+
+ // Construct a file stream from the above saved data
+ JavaPairDStream<LongWritable, IntWritable> testRaw = ssc.fileStream(
+ tempDir.getAbsolutePath() + "/" , SequenceFileInputFormat.class, LongWritable.class,
+ IntWritable.class, false);
+ JavaPairDStream<Long, Integer> test = testRaw.mapToPair(
+ new PairFunction<Tuple2<LongWritable, IntWritable>, Long, Integer>() {
+ public Tuple2<Long, Integer> call(Tuple2<LongWritable, IntWritable> input) {
+ return new Tuple2(input._1().get(), input._2().get());
+ }
+ });
+ final Accumulator<Integer> elem = ssc.sc().intAccumulator(0);
+ final Accumulator<Integer> total = ssc.sc().intAccumulator(0);
+ final Accumulator<Integer> calls = ssc.sc().intAccumulator(0);
+ test.foreachRDD(new Function<JavaPairRDD<Long, Integer>, Void>() {
+ public Void call(JavaPairRDD<Long, Integer> rdd) {
+ rdd.foreach(new VoidFunction<Tuple2<Long, Integer>>() {
+ public void call(Tuple2<Long, Integer> e) {
+ if (e._1() == 1l) {
+ elem.add(1);
+ }
+ total.add(1);
+ }
+ });
+ calls.add(1);
+ return null;
+ }
+ });
+ ssc.start();
+ Thread.sleep(5000);
--- End diff --
Could you make this something like a [`eventually`](http://doc.scalatest.org/1.8/org/scalatest/concurrent/Eventually.html) block in ScalaTest
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58456889
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21507/consoleFull) for PR 2703 at commit [`80704c5`](https://github.com/apache/spark/commit/80704c5dcc35252a9db1fdd39031b7376f92a7fa).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: WIP: Spark 3754 spark streaming file system ap...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58290329
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21433/Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-60323993
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22090/consoleFull) for PR 2703 at commit [`2bc3687`](https://github.com/apache/spark/commit/2bc368712918163f40afca3505d5e85b20fdd73d).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58451973
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21496/consoleFull) for PR 2703 at commit [`47cfdd5`](https://github.com/apache/spark/commit/47cfdd58075445822579c9271ef12521a7565ab0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r19316400
--- Diff: streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -1703,6 +1710,65 @@ public void testTextFileStream() {
JavaDStream<String> test = ssc.textFileStream("/tmp/foo");
}
+
+ @Test
+ public void testFileStream() throws Exception {
+ // Disable manual clock as FileInputDStream does not work with manual clock
+ System.setProperty("spark.streaming.clock", "org.apache.spark.streaming.util.SystemClock");
+ ssc = new JavaStreamingContext("local[2]", "test", new Duration(1000));
+ ssc.checkpoint("checkpoint");
+ // Set up some sequence files for streaming to read in
+ List<Tuple2<Long, Integer>> test_input = new ArrayList<Tuple2<Long, Integer> >();
+ test_input.add(new Tuple2(1L, 123456));
+ test_input.add(new Tuple2(2L, 123456));
+ JavaPairRDD<Long, Integer> rdd = ssc.sc().parallelizePairs(test_input);
+ File tempDir = Files.createTempDir();
+ JavaPairRDD<LongWritable, IntWritable> saveable = rdd.mapToPair(
+ new PairFunction<Tuple2<Long, Integer>, LongWritable, IntWritable>() {
+ public Tuple2<LongWritable, IntWritable> call(Tuple2<Long, Integer> record) {
+ return new Tuple2(new LongWritable(record._1), new IntWritable(record._2));
+ }});
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/1/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/2/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+
+ // Construct a file stream from the above saved data
+ JavaPairDStream<LongWritable, IntWritable> testRaw = ssc.fileStream(
+ tempDir.getAbsolutePath() + "/" , SequenceFileInputFormat.class, LongWritable.class,
+ IntWritable.class, false);
+ JavaPairDStream<Long, Integer> test = testRaw.mapToPair(
+ new PairFunction<Tuple2<LongWritable, IntWritable>, Long, Integer>() {
+ public Tuple2<Long, Integer> call(Tuple2<LongWritable, IntWritable> input) {
+ return new Tuple2(input._1().get(), input._2().get());
+ }
+ });
+ final Accumulator<Integer> elem = ssc.sc().intAccumulator(0);
--- End diff --
Cant this be made simpler, with DStream.count instead of tying in accumulators?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58456892
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21507/Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58447025
@tdas this fixes the issue with the Java API I found while working on the book.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-60316893
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22090/consoleFull) for PR 2703 at commit [`2bc3687`](https://github.com/apache/spark/commit/2bc368712918163f40afca3505d5e85b20fdd73d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58572814
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-62225023
Jenkins, test this again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk closed the pull request at:
https://github.com/apache/spark/pull/2703
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-61897978
@tdas: Ok will do. No one should be using the old API because it ins't callable from Java.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-61910562
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22958/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-62352321
[Test build #23146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23146/consoleFull) for PR 2703 at commit [`efac1fc`](https://github.com/apache/spark/commit/efac1fc5f2f46762ab339838b5c1a75707182e4a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-58453326
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21507/consoleFull) for PR 2703 at commit [`80704c5`](https://github.com/apache/spark/commit/80704c5dcc35252a9db1fdd39031b7376f92a7fa).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-61910555
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22958/consoleFull) for PR 2703 at commit [`3a2cd21`](https://github.com/apache/spark/commit/3a2cd21ec6672ecfb780d3a98fc5c1b5907c12e2).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-60113226
jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r19435781
--- Diff: streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -1703,6 +1710,65 @@ public void testTextFileStream() {
JavaDStream<String> test = ssc.textFileStream("/tmp/foo");
}
+
+ @Test
+ public void testFileStream() throws Exception {
+ // Disable manual clock as FileInputDStream does not work with manual clock
+ System.setProperty("spark.streaming.clock", "org.apache.spark.streaming.util.SystemClock");
+ ssc = new JavaStreamingContext("local[2]", "test", new Duration(1000));
+ ssc.checkpoint("checkpoint");
+ // Set up some sequence files for streaming to read in
+ List<Tuple2<Long, Integer>> test_input = new ArrayList<Tuple2<Long, Integer> >();
+ test_input.add(new Tuple2(1L, 123456));
+ test_input.add(new Tuple2(2L, 123456));
+ JavaPairRDD<Long, Integer> rdd = ssc.sc().parallelizePairs(test_input);
+ File tempDir = Files.createTempDir();
+ JavaPairRDD<LongWritable, IntWritable> saveable = rdd.mapToPair(
+ new PairFunction<Tuple2<Long, Integer>, LongWritable, IntWritable>() {
+ public Tuple2<LongWritable, IntWritable> call(Tuple2<Long, Integer> record) {
+ return new Tuple2(new LongWritable(record._1), new IntWritable(record._2));
+ }});
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/1/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+ saveable.saveAsNewAPIHadoopFile(tempDir.getAbsolutePath()+"/2/",
+ LongWritable.class, IntWritable.class,
+ SequenceFileOutputFormat.class);
+
+ // Construct a file stream from the above saved data
+ JavaPairDStream<LongWritable, IntWritable> testRaw = ssc.fileStream(
+ tempDir.getAbsolutePath() + "/" , SequenceFileInputFormat.class, LongWritable.class,
+ IntWritable.class, false);
+ JavaPairDStream<Long, Integer> test = testRaw.mapToPair(
+ new PairFunction<Tuple2<LongWritable, IntWritable>, Long, Integer>() {
+ public Tuple2<Long, Integer> call(Tuple2<LongWritable, IntWritable> input) {
+ return new Tuple2(input._1().get(), input._2().get());
+ }
+ });
+ final Accumulator<Integer> elem = ssc.sc().intAccumulator(0);
--- End diff --
How should I collect it back if I use DStream.count()? The provided utils for doing this in the scala tests didn't seem to match this very well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-60316196
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-62360203
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23146/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2703#issuecomment-60324001
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22090/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Spark 3754 spark streaming file system api cal...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/2703#discussion_r20126448
--- Diff: project/MimaExcludes.scala ---
@@ -78,6 +78,12 @@ object MimaExcludes {
ProblemFilters.exclude[IncompatibleResultTypeProblem](
"org.apache.spark.SparkContext.org$apache$spark$SparkContext$$createTaskScheduler")
) ++ Seq(
+ // Making Java Spark Streaming callable from Java
+ ProblemFilters.exclude[MissingMethodProblem](
--- End diff --
Please dont exclude StreamingContext related stuff. That should not change. Only the JavaStreamingContext should be affected.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org