You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zhangxinyu1 <gi...@git.apache.org> on 2016/10/14 11:03:23 UTC
[GitHub] spark pull request #15483: Add KafkaForeachWriter in external kafka-0.8.0 fo...
GitHub user zhangxinyu1 opened a pull request:
https://github.com/apache/spark/pull/15483
Add KafkaForeachWriter in external kafka-0.8.0 for structured streaming module
## What changes were proposed in this pull request?
I propose a `KafkaForeachWriter.scala` in external kafka-0.8.0.
It's useful for someone who requires to write results to kafka in structured streaming module. It can be used with `ForeachSink`, and the usage is as below:
`val query = input.writeStream`
`.outputMode("append")`
`.foreach(new KafkaForeachWriter[Row](producerProps, topics, input.schema)`
`.start()`
## How was this patch tested?
Unit test in `KafkaForeachWriterSuite.scala`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangxinyu1/spark feature-kafkaForeachWriter-in-external-kafka-0-8-for-structured-streaming
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15483.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15483
----
commit 8c2022755c08828edea3ef61f35a8490d1d6af24
Author: zhangxinyu1 <34...@qq.com>
Date: 2016-10-11T09:15:09Z
add KafkaForeachWriter for structured streaming. It can be used with ForeachSink
commit 1ad774919598aca5cf126c2c6e19fa618eaaa5d1
Author: zhangxinyu1 <34...@qq.com>
Date: 2016-10-14T10:29:33Z
Add KafkaForeachWriter to write result to kafka in structured streaming
It can be used with ForeachSink
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in exter...
Posted by zhangxinyu1 <gi...@git.apache.org>.
Github user zhangxinyu1 closed the pull request at:
https://github.com/apache/spark/pull/15483
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by zhangxinyu1 <gi...@git.apache.org>.
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/spark/pull/15483
@marmbrus
Yes, I also think `KafkaSink` is better than `KafkaForeachWriter` for users. I 'm trying to complete `KafkaSink` these days. Like `KafkaSource` in external kafka-0-10-sql, there is a procuer cache in executor to store kafkaProducers, which avoids repeatly creating and closing producers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15483
Thanks for working on this! However, I'm not sure that this is something that we should merge into the core repository (Though I think its an awesome example of how to use the `ForeachWriter` interface! You should consider making a blog post or even a spark package). Specifically, when we add support for writing to kafka, I think we'll want to do it both through the DataSource API, as well as the `Sink` API. This would let users seamlessly write batch and streaming jobs to Kafka in java, scala, python, R and SQL using the same code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15483
It would be good to post on the design / interfaces before you get too far.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15483
I will try to take a look soon. Can we close this PR until we have an updated implementation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by zhangxinyu1 <gi...@git.apache.org>.
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/spark/pull/15483
Sure. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by zhangxinyu1 <gi...@git.apache.org>.
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/spark/pull/15483
@marmbrus Could you take a look? Please tell me if you hava any suggestions?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by zhangxinyu1 <gi...@git.apache.org>.
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/spark/pull/15483
@marmbrus
I write a short deasign doc of KafkaSink in jira https://issues.apache.org/jira/browse/SPARK-17935
Could you please take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15483: [SPARK-17935][SQL]Add KafkaForeachWriter in external kaf...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15483
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org