You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by guoxiaolongzte <gi...@git.apache.org> on 2017/04/09 10:58:16 UTC
[GitHub] spark pull request #17580: [20269][Structured Streaming][Examples] add JavaW...
GitHub user guoxiaolongzte opened a pull request:
https://github.com/apache/spark/pull/17580
[20269][Structured Streaming][Examples] add JavaWordCountProducer in steaming examples
## What changes were proposed in this pull request?
run example of streaming kafka,currently missing java word count producer,not conducive to java developers to learn and test.
add a class JavaKafkaWordCountProducer
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guoxiaolongzte/spark SPARK-20269
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17580.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17580
----
commit d383efba12c66addb17006dea107bb0421d50bc3
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-03-31T13:57:09Z
[SPARK-20177]Document about compression way has some little detail changes.
commit 3059013e9d2aec76def14eb314b6761bea0e7ca0
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-01T01:38:02Z
[SPARK-20177] event log add a space
commit 46bb1ad3ddd9fb55b5607ac4f20213a90186cfe9
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T03:16:50Z
Merge branch 'master' of https://github.com/apache/spark into SPARK-20177
commit 0efb0dd9e404229cce638fe3fb0c966276784df7
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T03:47:53Z
[SPARK-20218]'/applications/[app-id]/stages' in REST API,add description.
commit 0e37fdeee28e31fc97436dabd001d3c85c5a7794
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T05:22:54Z
[SPARK-20218] '/applications/[app-id]/stages/[stage-id]' in REST API,remove redundant description.
commit 52641bb01e55b48bd9e8579fea217439d14c7dc7
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-07T06:24:58Z
Merge branch 'SPARK-20218'
commit d3977c9cab0722d279e3fae7aacbd4eb944c22f6
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-08T07:13:02Z
Merge branch 'master' of https://github.com/apache/spark
commit 1279041f17f1f5d02c9aada5fb5af9c7d58f2423
Author: asmith26 <as...@users.noreply.github.com>
Date: 2017-04-09T06:47:23Z
[MINOR] Issue: Change "slice" vs "partition" in exception messages (and code?)
## What changes were proposed in this pull request?
Came across the term "slice" when running some spark scala code. Consequently, a Google search indicated that "slices" and "partitions" refer to the same things; indeed see:
- [This issue](https://issues.apache.org/jira/browse/SPARK-1701)
- [This pull request](https://github.com/apache/spark/pull/2305)
- [This StackOverflow answer](http://stackoverflow.com/questions/23436640/what-is-the-difference-between-an-rdd-partition-and-a-slice) and [this one](http://stackoverflow.com/questions/24269495/what-are-the-differences-between-slices-and-partitions-of-rdds)
Thus this pull request fixes the occurrence of slice I came accross. Nonetheless, [it would appear](https://github.com/apache/spark/search?utf8=%E2%9C%93&q=slice&type=) there are still many references to "slice/slices" - thus I thought I'd raise this Pull Request to address the issue (sorry if this is the wrong place, I'm not too familar with raising apache issues).
## How was this patch tested?
(Not tested locally - only a minor exception message change.)
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: asmith26 <as...@users.noreply.github.com>
Closes #17565 from asmith26/master.
commit b0c7b1dcacc25f3ddf5c6d27b84a6e02122dbb73
Author: Sean Owen <so...@cloudera.com>
Date: 2017-04-09T07:44:02Z
[SPARK-19991][CORE][YARN] FileSegmentManagedBuffer performance improvement
## What changes were proposed in this pull request?
Avoid `NoSuchElementException` every time `ConfigProvider.get(val, default)` falls back to default. This apparently causes non-trivial overhead in at least one path, and can easily be avoided.
See https://github.com/apache/spark/pull/17329
## How was this patch tested?
Existing tests
Author: Sean Owen <so...@cloudera.com>
Closes #17567 from srowen/SPARK-19991.
commit 4c4e1c11fb326e446b2b4f668edf96876db5b187
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T07:48:34Z
Revert "[MINOR] Issue: Change "slice" vs "partition" in exception messages (and code?)"
This reverts commit 34fc48fb5976ede00f3f6d8c4d3eec979e4f4d7f.
commit d7ab5c15650686d63dbea568b7e0881eb804d015
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T10:55:06Z
Merge branch 'master' of https://github.com/apache/spark
commit fe3444df5311cff2092b02e5249310153ac8a060
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T10:56:15Z
add JavaWordCountProducer in steaming examples
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
@jerryshao
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
The above three classes in kafka-clients-0.10.0.1.jar\u3002The example here will not become obsolete.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
When a user use spark to develop a stream&kafka application, he first wants to find and learn example program in 'spark \ examples \ src \ main \ java \ org \ apache \ spark \ examples \ streaming' directory, will not see Scala directory any example of the program, so the example program has limited help to him. If you add a Java JavaKafkaWordCountProducer, for him to completely solve the problem.
Because since this class is only for the test convenience ,so i think it has direct relation to Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17580: [20269][Structured Streaming] add class 'JavaWord...
Posted by guoxiaolongzte <gi...@git.apache.org>.
GitHub user guoxiaolongzte reopened a pull request:
https://github.com/apache/spark/pull/17580
[20269][Structured Streaming] add class 'JavaWordCountProducer' to 'provide java word count producer'.
## What changes were proposed in this pull request?
1.run example of streaming kafka,currently missing java word count producer,not conducive to java developers to learn and test.
add a class JavaKafkaWordCountProducer.
2.run example of JavaKafkaWordCount.I find no java word count producer.
run example of KafkaWordCount.I find have scala word count producer.
I think we should provide the corresponding example code to facilitate java developers to learn and test.
3.My project team develops spark applications,basically with java statements and java API.
4.test case
1)java kafka word count producer,generate data for run example of JavaKafkaWordCount.the class of JavaKafkaWordCountProducer add by me.
spark-submit ... --class org.apache.spark.examples.streaming.JavaKafkaWordCountProducer --jars /home/gxl/spark/libext/kafka-clients-0.8.2.1.jar /home/gxl/spark/examples/jars/spark-examples_2.11-2.1.0.jar localhost:9092 topic1 3 4
2)java kafka streaming example
spark-submit ... --class org.apache.spark.examples.streaming.JavaKafkaWordCount --jars /home/gxl/kafka/kafka_2.11-0.10.2.0/libs/kafka-clients-0.10.2.0.jar --jars /home/gxl/spark/libext/kafka_2.11-0.8.2.1.jar /home/gxl/spark/examples/jars/spark-examples_2.11-2.1.0.jar localhost:2181 topic1 topic1 2
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guoxiaolongzte/spark SPARK-20269
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17580.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17580
----
commit d383efba12c66addb17006dea107bb0421d50bc3
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-03-31T13:57:09Z
[SPARK-20177]Document about compression way has some little detail changes.
commit 3059013e9d2aec76def14eb314b6761bea0e7ca0
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-01T01:38:02Z
[SPARK-20177] event log add a space
commit 46bb1ad3ddd9fb55b5607ac4f20213a90186cfe9
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T03:16:50Z
Merge branch 'master' of https://github.com/apache/spark into SPARK-20177
commit 0efb0dd9e404229cce638fe3fb0c966276784df7
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T03:47:53Z
[SPARK-20218]'/applications/[app-id]/stages' in REST API,add description.
commit 0e37fdeee28e31fc97436dabd001d3c85c5a7794
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T05:22:54Z
[SPARK-20218] '/applications/[app-id]/stages/[stage-id]' in REST API,remove redundant description.
commit 52641bb01e55b48bd9e8579fea217439d14c7dc7
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-07T06:24:58Z
Merge branch 'SPARK-20218'
commit d3977c9cab0722d279e3fae7aacbd4eb944c22f6
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Date: 2017-04-08T07:13:02Z
Merge branch 'master' of https://github.com/apache/spark
commit 1279041f17f1f5d02c9aada5fb5af9c7d58f2423
Author: asmith26 <as...@users.noreply.github.com>
Date: 2017-04-09T06:47:23Z
[MINOR] Issue: Change "slice" vs "partition" in exception messages (and code?)
## What changes were proposed in this pull request?
Came across the term "slice" when running some spark scala code. Consequently, a Google search indicated that "slices" and "partitions" refer to the same things; indeed see:
- [This issue](https://issues.apache.org/jira/browse/SPARK-1701)
- [This pull request](https://github.com/apache/spark/pull/2305)
- [This StackOverflow answer](http://stackoverflow.com/questions/23436640/what-is-the-difference-between-an-rdd-partition-and-a-slice) and [this one](http://stackoverflow.com/questions/24269495/what-are-the-differences-between-slices-and-partitions-of-rdds)
Thus this pull request fixes the occurrence of slice I came accross. Nonetheless, [it would appear](https://github.com/apache/spark/search?utf8=%E2%9C%93&q=slice&type=) there are still many references to "slice/slices" - thus I thought I'd raise this Pull Request to address the issue (sorry if this is the wrong place, I'm not too familar with raising apache issues).
## How was this patch tested?
(Not tested locally - only a minor exception message change.)
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: asmith26 <as...@users.noreply.github.com>
Closes #17565 from asmith26/master.
commit b0c7b1dcacc25f3ddf5c6d27b84a6e02122dbb73
Author: Sean Owen <so...@cloudera.com>
Date: 2017-04-09T07:44:02Z
[SPARK-19991][CORE][YARN] FileSegmentManagedBuffer performance improvement
## What changes were proposed in this pull request?
Avoid `NoSuchElementException` every time `ConfigProvider.get(val, default)` falls back to default. This apparently causes non-trivial overhead in at least one path, and can easily be avoided.
See https://github.com/apache/spark/pull/17329
## How was this patch tested?
Existing tests
Author: Sean Owen <so...@cloudera.com>
Closes #17567 from srowen/SPARK-19991.
commit 4c4e1c11fb326e446b2b4f668edf96876db5b187
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T07:48:34Z
Revert "[MINOR] Issue: Change "slice" vs "partition" in exception messages (and code?)"
This reverts commit 34fc48fb5976ede00f3f6d8c4d3eec979e4f4d7f.
commit d7ab5c15650686d63dbea568b7e0881eb804d015
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T10:55:06Z
Merge branch 'master' of https://github.com/apache/spark
commit fe3444df5311cff2092b02e5249310153ac8a060
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T10:56:15Z
add JavaWordCountProducer in steaming examples
commit 6f0491d4f1c5647520aaebb75152ff0a8e38e8ef
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-09T12:43:14Z
modify java code style.Use 2-space indentation in general
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17580: [20269][Structured Streaming] add class 'JavaWord...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17580#discussion_r110540044
--- Diff: examples/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCountProducer.java ---
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.streaming;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+
+/**
+ * words producer to java kafka word count
+ * Produces some random words between 1 and 10
+ * Usage: JavaKafkaWordCountProducer <metadataBrokerList> <topic> <messagesPerSec> <wordsPerMessage>
+ * <metadataBrokerList> is a list of one or more kafka broker servers
+ * <topic> is a kafka topic to consume from
+ * <messagesPerSec> is the number of messages generated per second
+ * <wordsPerMessage> is the number of words per message
+ */
+public class JavaKafkaWordCountProducer {
+ public static void main(String[] args) {
+ if (args.length < 4) {
--- End diff --
We follow 2 space indentation for Java. Please review "Code Style Guide" in http://spark.apache.org/contributing.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17580
I would say this is not a Spark program, it is absolutely a Kafka producer code. To maintain a Kafka Producer example in Spark is not a good choice, this is a legacy code. Because of the API changes of Kafka, the example here will become obsolete and maintenance overhead (I fixed the deprecated APIs here once shifting to 0.8.2). So instead of adding Java counterpart, I would rather to remove it. (Just my two cents).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17580
I think I'd go with @jerryshao and not add it if it's an old, auxiliary example anyway
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
@Override
public String call(Tuple2<String, String> tuple2) {
return tuple2._2();
}
});
JavaDStream<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
@Override
public Iterator<String> call(String x) {
return Arrays.asList(SPACE.split(x)).iterator();
}
});
JavaPairDStream<String, Integer> wordCounts = words.mapToPair(
new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String s) {
return new Tuple2<>(s, 1);
}
}).reduceByKey(new Function2<Integer, Integer, Integer>() {
@Override
public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
**The above code version in 2.1.0 release.It is only recently changed to Scala grammar in apache master. I want to know why.Thank you!**@jerryshao
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming][Examples] add JavaWordCoun...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17580
The code style needs to be fixed, and the title. What example is this based on ? this is the kind of info that you should put in a pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
@srowen
@jerryshao
I understand, thank you.
I changed the code in my project to keep the program consistent with the latest example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
Title,PR description and motive, has been modified.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17580
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17580
It is just Java8 lambda function, nothing related to Scala...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
Because of the API changes of Kafka, we do not want to delete it, but to maintain and modify it.
Although it is absolutely a Kafka producer code, but this is part of spark streaming, it is also very important.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17580
In some cases you have to call a Scala API from Java and need to create Scala-specific classes like Tuple2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming][Examples] add JavaWordCoun...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17580
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
Sorry,spark java code style is different from the style of my project team.Now I know, and have been fixed.
Use 2-space indentation in general. For function declarations, use 4 space indentation for its parameters when they don\u2019t fit in a single line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17580
What's the meaning of "some of the Scala"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
@jerryshao
Excuse me,I would like to ask, these examples of Java examples, why use some of the Scala to write it?thank you!
SparkConf sparkConf = new SparkConf().setAppName("**JavaKafkaWordCount**");
// Create the context with 2 seconds batch size
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(2000));
int numThreads = Integer.parseInt(args[3]);
Map<String, Integer> topicMap = new HashMap<>();
String[] topics = args[2].split(",");
for (String topic: topics) {
topicMap.put(topic, numThreads);
}
JavaPairReceiverInputDStream<String, String> messages =
KafkaUtils.createStream(jssc, args[0], args[1], topicMap);
JavaDStream<String> lines = messages.map**(Tuple2::_2);**
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17580
You mean this [line](https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaKafkaWordCount.java#L76)?
It's because our KafkInputDStream signature is `ReceiverInputDStream[(K, V)]`, in which the Type is `(K, V)`, this `(K, V)` is Scala's Tuple2. That's why in the Java code we need to call Scala's API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17580: [20269][Structured Streaming] add class 'JavaWord...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte closed the pull request at:
https://github.com/apache/spark/pull/17580
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17580: [20269][Structured Streaming] add class 'JavaWord...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte closed the pull request at:
https://github.com/apache/spark/pull/17580
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17580
I think this is not worth bothering with for the reasons above, and can be closed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
![Uploading image.png\u2026]()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17580
What is the purpose of adding this example? I think we already have a `KafkaWordCountProducer` for the convenience of Kafka streaming example, and we could use that to send events to Kafka. I think there's no need to add a Java counterpart, since this class is only for the test convenience and has no direct relation to Spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17580: [20269][Structured Streaming] add class 'JavaWordCountPr...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17580
eg
messages.map(Tuple2::_2) is the Scala grammar code.But JavaKafkaWordCount is a java class.
I think the Java example program should be fully implemented in Java code.
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org