You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zsxwing <gi...@git.apache.org> on 2016/10/06 00:03:28 UTC
[GitHub] spark pull request #15367: [SPARK-17346][SQL] Add Kafka source for Structure...
GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/15367
[SPARK-17346][SQL] Add Kafka source for Structured Streaming (branch 2.0)
## What changes were proposed in this pull request?
Backport https://github.com/apache/spark/commit/9293734d35eb3d6e4fd4ebb86f54dd5d3a35e6db into branch 2.0.
The only difference is the Spark version in pom file.
## How was this patch tested?
Jenkins.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark kafka-source-branch-2.0
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15367.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15367
----
commit 17e6b5c3fb702e66748634260670f4f843aa38fe
Author: Shixiong Zhu <sh...@databricks.com>
Date: 2016-10-05T23:45:45Z
[SPARK-17346][SQL] Add Kafka source for Structured Streaming
## What changes were proposed in this pull request?
This PR adds a new project ` external/kafka-0-10-sql` for Structured Streaming Kafka source.
It's based on the design doc: https://docs.google.com/document/d/19t2rWe51x7tq2e5AOfrsM9qb8_m7BRuv9fel9i0PqR8/edit?usp=sharing
tdas did most of work and part of them was inspired by koeninger's work.
### Introduction
The Kafka source is a structured streaming data source to poll data from Kafka. The schema of reading data is as follows:
Column | Type
---- | ----
key | binary
value | binary
topic | string
partition | int
offset | long
timestamp | long
timestampType | int
The source can deal with deleting topics. However, the user should make sure there is no Spark job processing the data when deleting a topic.
### Configuration
The user can use `DataStreamReader.option` to set the following configurations.
Kafka Source's options | value | default | meaning
------ | ------- | ------ | -----
startingOffset | ["earliest", "latest"] | "latest" | The start point when a query is started, either "earliest" which is from the earliest offset, or "latest" which is just from the latest offset. Note: This only applies when a new Streaming query is started, and that resuming will always pick up from where the query left off.
failOnDataLost | [true, false] | true | Whether to fail the query when it's possible that data is lost (e.g., topics are deleted, or offsets are out of range). This may be a false alarm. You can disable it when it doesn't work as you expected.
subscribe | A comma-separated list of topics | (none) | The topic list to subscribe. Only one of "subscribe" and "subscribeParttern" options can be specified for Kafka source.
subscribePattern | Java regex string | (none) | The pattern used to subscribe the topic. Only one of "subscribe" and "subscribeParttern" options can be specified for Kafka source.
kafka.consumer.poll.timeoutMs | long | 512 | The timeout in milliseconds to poll data from Kafka in executors
fetchOffset.numRetries | int | 3 | Number of times to retry before giving up fatch Kafka latest offsets.
fetchOffset.retryIntervalMs | long | 10 | milliseconds to wait before retrying to fetch Kafka offsets
Kafka's own configurations can be set via `DataStreamReader.option` with `kafka.` prefix, e.g, `stream.option("kafka.bootstrap.servers", "host:port")`
### Usage
* Subscribe to 1 topic
```Scala
spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host:port")
.option("subscribe", "topic1")
.load()
```
* Subscribe to multiple topics
```Scala
spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host:port")
.option("subscribe", "topic1,topic2")
.load()
```
* Subscribe to a pattern
```Scala
spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "host:port")
.option("subscribePattern", "topic.*")
.load()
```
## How was this patch tested?
The new unit tests.
Author: Shixiong Zhu <sh...@databricks.com>
Author: Tathagata Das <ta...@gmail.com>
Author: Shixiong Zhu <zs...@gmail.com>
Author: cody koeninger <co...@koeninger.org>
Closes #15102 from zsxwing/kafka-source.
commit a560190dba618698ae2d9728e8b35c5954772467
Author: Shixiong Zhu <sh...@databricks.com>
Date: 2016-10-06T00:02:12Z
Fix version
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15367
@holdenk this is basically adding an external project. Without the Kafka source, it's hard for people to test Structured Streaming with their real data. Considering 2.0.2 will be out before 2.1.0, it's better to add this one into branch 2.0 in order to test it early. Since Structured Streaming is experimental, it won't break any production usage and should be safe.
/cc @rxin @marmbrus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/15367
This is rather a lot of functionality to backport, our backport guide doesn't seem to quite line up with this (based on https://www.mail-archive.com/dev@spark.apache.org/msg10284.html / https://cwiki.apache.org/confluence/display/SPARK/Committers#Committers-PolicyonBackportingBugFixes ) but @pwendell will know better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66408/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66423/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15367: [SPARK-17346][SQL][test-maven]Add Kafka source fo...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing closed the pull request at:
https://github.com/apache/spark/pull/15367
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15367
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by koeninger <gi...@git.apache.org>.
Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15367
Does backporting reduce the likelihood of change if user feedback indicates we got it wrong?
My technical concerns were largely addressed, that's my big remaining organizational concern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15367
We should definitly vet this PR carefully to make sure its safe. One thing that is missing from that guide, that I do believe is accepted practice, is more leeway when the feature is marked experimental (i.e. during Spark 1.0-1.3 I used to backport pretty much everything that went into SQL). Since I think many people are simply not using structured streaming due to lack of kafka support, I'd error on the side of more users sooner. If you think any part of this PR is unsafe with respect stable components though, please do raise those concerns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15367: [SPARK-17346][SQL][test-maven]Add Kafka source fo...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/15367#discussion_r82313748
--- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.kafka010
+
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.util.Random
+
+import org.apache.kafka.clients.producer.RecordMetadata
+import org.scalatest.BeforeAndAfter
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.sql.execution.streaming._
+import org.apache.spark.sql.streaming.StreamTest
+import org.apache.spark.sql.test.SharedSQLContext
+
+
+abstract class KafkaSourceTest extends StreamTest with SharedSQLContext {
+
+ protected var testUtils: KafkaTestUtils = _
+
+ override val streamingTimeout = 30.seconds
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ testUtils = new KafkaTestUtils
+ testUtils.setup()
+ }
+
+ override def afterAll(): Unit = {
+ if (testUtils != null) {
+ testUtils.teardown()
+ testUtils = null
+ super.afterAll()
+ }
+ }
+
+ protected def makeSureGetOffsetCalled = AssertOnQuery { q =>
+ // Because KafkaSource's initialPartitionOffsets is set lazily, we need to make sure
+ // its "getOffset" is called before pushing any data. Otherwise, because of the race contion,
+ // we don't know which data should be fetched when `startingOffset` is latest.
+ q.processAllAvailable()
+ true
+ }
+
+ /**
+ * Add data to Kafka.
+ *
+ * `topicAction` can be used to run actions for each topic before inserting data.
+ */
+ case class AddKafkaData(topics: Set[String], data: Int*)
+ (implicit ensureDataInMultiplePartition: Boolean = false,
+ concurrent: Boolean = false,
+ message: String = "",
+ topicAction: (String, Option[Int]) => Unit = (_, _) => {}) extends AddData {
+
+ override def addData(query: Option[StreamExecution]): (Source, Offset) = {
+ if (query.get.isActive) {
+ // Make sure no Spark job is running when deleting a topic
+ query.get.processAllAvailable()
+ }
+
+ val existingTopics = testUtils.getAllTopicsAndPartitionSize().toMap
+ val newTopics = topics.diff(existingTopics.keySet)
+ for (newTopic <- newTopics) {
+ topicAction(newTopic, None)
+ }
+ for (existingTopicPartitions <- existingTopics) {
+ topicAction(existingTopicPartitions._1, Some(existingTopicPartitions._2))
+ }
+
+ // Read all topics again in case some topics are delete.
+ val allTopics = testUtils.getAllTopicsAndPartitionSize().toMap.keys
+ require(
+ query.nonEmpty,
+ "Cannot add data when there is no query for finding the active kafka source")
+
+ val sources = query.get.logicalPlan.collect {
+ case StreamingExecutionRelation(source, _) if source.isInstanceOf[KafkaSource] =>
+ source.asInstanceOf[KafkaSource]
+ }
+ if (sources.isEmpty) {
+ throw new Exception(
+ "Could not find Kafka source in the StreamExecution logical plan to add data to")
+ } else if (sources.size > 1) {
+ throw new Exception(
+ "Could not select the Kafka source in the StreamExecution logical plan as there" +
+ "are multiple Kafka sources:\n\t" + sources.mkString("\n\t"))
+ }
+ val kafkaSource = sources.head
+ val topic = topics.toSeq(Random.nextInt(topics.size))
+ val sentMetadata = testUtils.sendMessages(topic, data.map { _.toString }.toArray)
+
+ def metadataToStr(m: (String, RecordMetadata)): String = {
+ s"Sent ${m._1} to partition ${m._2.partition()}, offset ${m._2.offset()}"
+ }
+ // Verify that the test data gets inserted into multiple partitions
+ if (ensureDataInMultiplePartition) {
+ require(
+ sentMetadata.groupBy(_._2.partition).size > 1,
+ s"Added data does not test multiple partitions: ${sentMetadata.map(metadataToStr)}")
+ }
+
+ val offset = KafkaSourceOffset(testUtils.getLatestOffsets(topics))
+ logInfo(s"Added data, expected offset $offset")
+ (kafkaSource, offset)
+ }
+
+ override def toString: String =
+ s"AddKafkaData(topics = $topics, data = $data, message = $message)"
+ }
+}
+
+
+class KafkaSourceSuite extends KafkaSourceTest {
+
+ import testImplicits._
+
+ private val topicId = new AtomicInteger(0)
+
+ test("cannot stop Kafka stream") {
+ val topic = newTopic()
+ testUtils.createTopic(newTopic(), partitions = 5)
+ testUtils.sendMessages(topic, (101 to 105).map { _.toString }.toArray)
+
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ .option("subscribePattern", s"topic-.*")
+
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ makeSureGetOffsetCalled,
+ StopStream
+ )
+ }
+
+ test("subscribing topic by name from latest offsets") {
+ val topic = newTopic()
+ testFromLatestOffsets(topic, "subscribe" -> topic)
+ }
+
+ test("subscribing topic by name from earliest offsets") {
+ val topic = newTopic()
+ testFromEarliestOffsets(topic, "subscribe" -> topic)
+ }
+
+ test("subscribing topic by pattern from latest offsets") {
+ val topicPrefix = newTopic()
+ val topic = topicPrefix + "-suffix"
+ testFromLatestOffsets(topic, "subscribePattern" -> s"$topicPrefix-.*")
+ }
+
+ test("subscribing topic by pattern from earliest offsets") {
+ val topicPrefix = newTopic()
+ val topic = topicPrefix + "-suffix"
+ testFromEarliestOffsets(topic, "subscribePattern" -> s"$topicPrefix-.*")
+ }
+
+ test("subscribing topic by pattern with topic deletions") {
+ val topicPrefix = newTopic()
+ val topic = topicPrefix + "-seems"
+ val topic2 = topicPrefix + "-bad"
+ testUtils.createTopic(topic, partitions = 5)
+ testUtils.sendMessages(topic, Array("-1"))
+ require(testUtils.getLatestOffsets(Set(topic)).size === 5)
+
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ .option("subscribePattern", s"$topicPrefix-.*")
+ .option("failOnDataLoss", "false")
+
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ makeSureGetOffsetCalled,
+ AddKafkaData(Set(topic), 1, 2, 3),
+ CheckAnswer(2, 3, 4),
+ Assert {
+ testUtils.deleteTopic(topic)
+ testUtils.createTopic(topic2, partitions = 5)
+ true
+ },
+ AddKafkaData(Set(topic2), 4, 5, 6),
+ CheckAnswer(2, 3, 4, 5, 6, 7)
+ )
+ }
+
+ test("bad source options") {
+ def testBadOptions(options: (String, String)*)(expectedMsgs: String*): Unit = {
+ val ex = intercept[IllegalArgumentException] {
+ val reader = spark
+ .readStream
+ .format("kafka")
+ options.foreach { case (k, v) => reader.option(k, v) }
+ reader.load()
+ }
+ expectedMsgs.foreach { m =>
+ assert(ex.getMessage.toLowerCase.contains(m.toLowerCase))
+ }
+ }
+
+ // No strategy specified
+ testBadOptions()("options must be specified", "subscribe", "subscribePattern")
+
+ // Multiple strategies specified
+ testBadOptions("subscribe" -> "t", "subscribePattern" -> "t.*")(
+ "only one", "options can be specified")
+
+ testBadOptions("subscribe" -> "")("no topics to subscribe")
+ testBadOptions("subscribePattern" -> "")("pattern to subscribe is empty")
+ }
+
+ test("unsupported kafka configs") {
+ def testUnsupportedConfig(key: String, value: String = "someValue"): Unit = {
+ val ex = intercept[IllegalArgumentException] {
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("subscribe", "topic")
+ .option("kafka.bootstrap.servers", "somehost")
+ .option(s"$key", value)
+ reader.load()
+ }
+ assert(ex.getMessage.toLowerCase.contains("not supported"))
+ }
+
+ testUnsupportedConfig("kafka.group.id")
+ testUnsupportedConfig("kafka.auto.offset.reset")
+ testUnsupportedConfig("kafka.enable.auto.commit")
+ testUnsupportedConfig("kafka.interceptor.classes")
+ testUnsupportedConfig("kafka.key.deserializer")
+ testUnsupportedConfig("kafka.value.deserializer")
+
+ testUnsupportedConfig("kafka.auto.offset.reset", "none")
+ testUnsupportedConfig("kafka.auto.offset.reset", "someValue")
+ testUnsupportedConfig("kafka.auto.offset.reset", "earliest")
+ testUnsupportedConfig("kafka.auto.offset.reset", "latest")
+ }
+
+ private def newTopic(): String = s"topic-${topicId.getAndIncrement()}"
+
+ private def testFromLatestOffsets(topic: String, options: (String, String)*): Unit = {
+ testUtils.createTopic(topic, partitions = 5)
+ testUtils.sendMessages(topic, Array("-1"))
+ require(testUtils.getLatestOffsets(Set(topic)).size === 5)
+
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("startingOffset", s"latest")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ options.foreach { case (k, v) => reader.option(k, v) }
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ makeSureGetOffsetCalled,
+ AddKafkaData(Set(topic), 1, 2, 3),
+ CheckAnswer(2, 3, 4),
+ StopStream,
+ StartStream(),
+ CheckAnswer(2, 3, 4), // Should get the data back on recovery
+ StopStream,
+ AddKafkaData(Set(topic), 4, 5, 6), // Add data when stream is stopped
+ StartStream(),
+ CheckAnswer(2, 3, 4, 5, 6, 7), // Should get the added data
+ AddKafkaData(Set(topic), 7, 8),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9),
+ AssertOnQuery("Add partitions") { query: StreamExecution =>
+ testUtils.addPartitions(topic, 10)
+ true
+ },
+ AddKafkaData(Set(topic), 9, 10, 11, 12, 13, 14, 15, 16),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
+ )
+ }
+
+ private def testFromEarliestOffsets(topic: String, options: (String, String)*): Unit = {
+ testUtils.createTopic(topic, partitions = 5)
+ testUtils.sendMessages(topic, (1 to 3).map { _.toString }.toArray)
+ require(testUtils.getLatestOffsets(Set(topic)).size === 5)
+
+ val reader = spark.readStream
+ reader
+ .format(classOf[KafkaSourceProvider].getCanonicalName.stripSuffix("$"))
+ .option("startingOffset", s"earliest")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ options.foreach { case (k, v) => reader.option(k, v) }
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ AddKafkaData(Set(topic), 4, 5, 6), // Add data when stream is stopped
+ CheckAnswer(2, 3, 4, 5, 6, 7),
+ StopStream,
+ StartStream(),
+ CheckAnswer(2, 3, 4, 5, 6, 7),
+ StopStream,
+ AddKafkaData(Set(topic), 7, 8),
+ StartStream(),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9),
+ AssertOnQuery("Add partitions") { query: StreamExecution =>
+ testUtils.addPartitions(topic, 10)
+ true
+ },
+ AddKafkaData(Set(topic), 9, 10, 11, 12, 13, 14, 15, 16),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
+ )
+ }
+}
+
+
+class KafkaSourceStressSuite extends KafkaSourceTest with BeforeAndAfter {
+
+ import testImplicits._
+
+ val topicId = new AtomicInteger(1)
+
+ @volatile var topics: Seq[String] = (1 to 5).map(_ => newStressTopic)
+
+ def newStressTopic: String = s"stress${topicId.getAndIncrement()}"
+
+ private def nextInt(start: Int, end: Int): Int = {
+ start + Random.nextInt(start + end - 1)
+ }
+
+ after {
+ for (topic <- testUtils.getAllTopicsAndPartitionSize().toMap.keys) {
--- End diff --
@tdas we can just merge this one then merge my another PR to 2.0. There should not be any conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66433/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15367
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66433/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15367: [SPARK-17346][SQL][test-maven]Add Kafka source fo...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/15367#discussion_r82313116
--- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceSuite.scala ---
@@ -0,0 +1,425 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.kafka010
+
+import java.util.concurrent.atomic.AtomicInteger
+
+import scala.util.Random
+
+import org.apache.kafka.clients.producer.RecordMetadata
+import org.scalatest.BeforeAndAfter
+import org.scalatest.time.SpanSugar._
+
+import org.apache.spark.sql.execution.streaming._
+import org.apache.spark.sql.streaming.StreamTest
+import org.apache.spark.sql.test.SharedSQLContext
+
+
+abstract class KafkaSourceTest extends StreamTest with SharedSQLContext {
+
+ protected var testUtils: KafkaTestUtils = _
+
+ override val streamingTimeout = 30.seconds
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ testUtils = new KafkaTestUtils
+ testUtils.setup()
+ }
+
+ override def afterAll(): Unit = {
+ if (testUtils != null) {
+ testUtils.teardown()
+ testUtils = null
+ super.afterAll()
+ }
+ }
+
+ protected def makeSureGetOffsetCalled = AssertOnQuery { q =>
+ // Because KafkaSource's initialPartitionOffsets is set lazily, we need to make sure
+ // its "getOffset" is called before pushing any data. Otherwise, because of the race contion,
+ // we don't know which data should be fetched when `startingOffset` is latest.
+ q.processAllAvailable()
+ true
+ }
+
+ /**
+ * Add data to Kafka.
+ *
+ * `topicAction` can be used to run actions for each topic before inserting data.
+ */
+ case class AddKafkaData(topics: Set[String], data: Int*)
+ (implicit ensureDataInMultiplePartition: Boolean = false,
+ concurrent: Boolean = false,
+ message: String = "",
+ topicAction: (String, Option[Int]) => Unit = (_, _) => {}) extends AddData {
+
+ override def addData(query: Option[StreamExecution]): (Source, Offset) = {
+ if (query.get.isActive) {
+ // Make sure no Spark job is running when deleting a topic
+ query.get.processAllAvailable()
+ }
+
+ val existingTopics = testUtils.getAllTopicsAndPartitionSize().toMap
+ val newTopics = topics.diff(existingTopics.keySet)
+ for (newTopic <- newTopics) {
+ topicAction(newTopic, None)
+ }
+ for (existingTopicPartitions <- existingTopics) {
+ topicAction(existingTopicPartitions._1, Some(existingTopicPartitions._2))
+ }
+
+ // Read all topics again in case some topics are delete.
+ val allTopics = testUtils.getAllTopicsAndPartitionSize().toMap.keys
+ require(
+ query.nonEmpty,
+ "Cannot add data when there is no query for finding the active kafka source")
+
+ val sources = query.get.logicalPlan.collect {
+ case StreamingExecutionRelation(source, _) if source.isInstanceOf[KafkaSource] =>
+ source.asInstanceOf[KafkaSource]
+ }
+ if (sources.isEmpty) {
+ throw new Exception(
+ "Could not find Kafka source in the StreamExecution logical plan to add data to")
+ } else if (sources.size > 1) {
+ throw new Exception(
+ "Could not select the Kafka source in the StreamExecution logical plan as there" +
+ "are multiple Kafka sources:\n\t" + sources.mkString("\n\t"))
+ }
+ val kafkaSource = sources.head
+ val topic = topics.toSeq(Random.nextInt(topics.size))
+ val sentMetadata = testUtils.sendMessages(topic, data.map { _.toString }.toArray)
+
+ def metadataToStr(m: (String, RecordMetadata)): String = {
+ s"Sent ${m._1} to partition ${m._2.partition()}, offset ${m._2.offset()}"
+ }
+ // Verify that the test data gets inserted into multiple partitions
+ if (ensureDataInMultiplePartition) {
+ require(
+ sentMetadata.groupBy(_._2.partition).size > 1,
+ s"Added data does not test multiple partitions: ${sentMetadata.map(metadataToStr)}")
+ }
+
+ val offset = KafkaSourceOffset(testUtils.getLatestOffsets(topics))
+ logInfo(s"Added data, expected offset $offset")
+ (kafkaSource, offset)
+ }
+
+ override def toString: String =
+ s"AddKafkaData(topics = $topics, data = $data, message = $message)"
+ }
+}
+
+
+class KafkaSourceSuite extends KafkaSourceTest {
+
+ import testImplicits._
+
+ private val topicId = new AtomicInteger(0)
+
+ test("cannot stop Kafka stream") {
+ val topic = newTopic()
+ testUtils.createTopic(newTopic(), partitions = 5)
+ testUtils.sendMessages(topic, (101 to 105).map { _.toString }.toArray)
+
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ .option("subscribePattern", s"topic-.*")
+
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ makeSureGetOffsetCalled,
+ StopStream
+ )
+ }
+
+ test("subscribing topic by name from latest offsets") {
+ val topic = newTopic()
+ testFromLatestOffsets(topic, "subscribe" -> topic)
+ }
+
+ test("subscribing topic by name from earliest offsets") {
+ val topic = newTopic()
+ testFromEarliestOffsets(topic, "subscribe" -> topic)
+ }
+
+ test("subscribing topic by pattern from latest offsets") {
+ val topicPrefix = newTopic()
+ val topic = topicPrefix + "-suffix"
+ testFromLatestOffsets(topic, "subscribePattern" -> s"$topicPrefix-.*")
+ }
+
+ test("subscribing topic by pattern from earliest offsets") {
+ val topicPrefix = newTopic()
+ val topic = topicPrefix + "-suffix"
+ testFromEarliestOffsets(topic, "subscribePattern" -> s"$topicPrefix-.*")
+ }
+
+ test("subscribing topic by pattern with topic deletions") {
+ val topicPrefix = newTopic()
+ val topic = topicPrefix + "-seems"
+ val topic2 = topicPrefix + "-bad"
+ testUtils.createTopic(topic, partitions = 5)
+ testUtils.sendMessages(topic, Array("-1"))
+ require(testUtils.getLatestOffsets(Set(topic)).size === 5)
+
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ .option("subscribePattern", s"$topicPrefix-.*")
+ .option("failOnDataLoss", "false")
+
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ makeSureGetOffsetCalled,
+ AddKafkaData(Set(topic), 1, 2, 3),
+ CheckAnswer(2, 3, 4),
+ Assert {
+ testUtils.deleteTopic(topic)
+ testUtils.createTopic(topic2, partitions = 5)
+ true
+ },
+ AddKafkaData(Set(topic2), 4, 5, 6),
+ CheckAnswer(2, 3, 4, 5, 6, 7)
+ )
+ }
+
+ test("bad source options") {
+ def testBadOptions(options: (String, String)*)(expectedMsgs: String*): Unit = {
+ val ex = intercept[IllegalArgumentException] {
+ val reader = spark
+ .readStream
+ .format("kafka")
+ options.foreach { case (k, v) => reader.option(k, v) }
+ reader.load()
+ }
+ expectedMsgs.foreach { m =>
+ assert(ex.getMessage.toLowerCase.contains(m.toLowerCase))
+ }
+ }
+
+ // No strategy specified
+ testBadOptions()("options must be specified", "subscribe", "subscribePattern")
+
+ // Multiple strategies specified
+ testBadOptions("subscribe" -> "t", "subscribePattern" -> "t.*")(
+ "only one", "options can be specified")
+
+ testBadOptions("subscribe" -> "")("no topics to subscribe")
+ testBadOptions("subscribePattern" -> "")("pattern to subscribe is empty")
+ }
+
+ test("unsupported kafka configs") {
+ def testUnsupportedConfig(key: String, value: String = "someValue"): Unit = {
+ val ex = intercept[IllegalArgumentException] {
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("subscribe", "topic")
+ .option("kafka.bootstrap.servers", "somehost")
+ .option(s"$key", value)
+ reader.load()
+ }
+ assert(ex.getMessage.toLowerCase.contains("not supported"))
+ }
+
+ testUnsupportedConfig("kafka.group.id")
+ testUnsupportedConfig("kafka.auto.offset.reset")
+ testUnsupportedConfig("kafka.enable.auto.commit")
+ testUnsupportedConfig("kafka.interceptor.classes")
+ testUnsupportedConfig("kafka.key.deserializer")
+ testUnsupportedConfig("kafka.value.deserializer")
+
+ testUnsupportedConfig("kafka.auto.offset.reset", "none")
+ testUnsupportedConfig("kafka.auto.offset.reset", "someValue")
+ testUnsupportedConfig("kafka.auto.offset.reset", "earliest")
+ testUnsupportedConfig("kafka.auto.offset.reset", "latest")
+ }
+
+ private def newTopic(): String = s"topic-${topicId.getAndIncrement()}"
+
+ private def testFromLatestOffsets(topic: String, options: (String, String)*): Unit = {
+ testUtils.createTopic(topic, partitions = 5)
+ testUtils.sendMessages(topic, Array("-1"))
+ require(testUtils.getLatestOffsets(Set(topic)).size === 5)
+
+ val reader = spark
+ .readStream
+ .format("kafka")
+ .option("startingOffset", s"latest")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ options.foreach { case (k, v) => reader.option(k, v) }
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ makeSureGetOffsetCalled,
+ AddKafkaData(Set(topic), 1, 2, 3),
+ CheckAnswer(2, 3, 4),
+ StopStream,
+ StartStream(),
+ CheckAnswer(2, 3, 4), // Should get the data back on recovery
+ StopStream,
+ AddKafkaData(Set(topic), 4, 5, 6), // Add data when stream is stopped
+ StartStream(),
+ CheckAnswer(2, 3, 4, 5, 6, 7), // Should get the added data
+ AddKafkaData(Set(topic), 7, 8),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9),
+ AssertOnQuery("Add partitions") { query: StreamExecution =>
+ testUtils.addPartitions(topic, 10)
+ true
+ },
+ AddKafkaData(Set(topic), 9, 10, 11, 12, 13, 14, 15, 16),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
+ )
+ }
+
+ private def testFromEarliestOffsets(topic: String, options: (String, String)*): Unit = {
+ testUtils.createTopic(topic, partitions = 5)
+ testUtils.sendMessages(topic, (1 to 3).map { _.toString }.toArray)
+ require(testUtils.getLatestOffsets(Set(topic)).size === 5)
+
+ val reader = spark.readStream
+ reader
+ .format(classOf[KafkaSourceProvider].getCanonicalName.stripSuffix("$"))
+ .option("startingOffset", s"earliest")
+ .option("kafka.bootstrap.servers", testUtils.brokerAddress)
+ .option("kafka.metadata.max.age.ms", "1")
+ options.foreach { case (k, v) => reader.option(k, v) }
+ val kafka = reader.load()
+ .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
+ .as[(String, String)]
+ val mapped = kafka.map(kv => kv._2.toInt + 1)
+
+ testStream(mapped)(
+ AddKafkaData(Set(topic), 4, 5, 6), // Add data when stream is stopped
+ CheckAnswer(2, 3, 4, 5, 6, 7),
+ StopStream,
+ StartStream(),
+ CheckAnswer(2, 3, 4, 5, 6, 7),
+ StopStream,
+ AddKafkaData(Set(topic), 7, 8),
+ StartStream(),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9),
+ AssertOnQuery("Add partitions") { query: StreamExecution =>
+ testUtils.addPartitions(topic, 10)
+ true
+ },
+ AddKafkaData(Set(topic), 9, 10, 11, 12, 13, 14, 15, 16),
+ CheckAnswer(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
+ )
+ }
+}
+
+
+class KafkaSourceStressSuite extends KafkaSourceTest with BeforeAndAfter {
+
+ import testImplicits._
+
+ val topicId = new AtomicInteger(1)
+
+ @volatile var topics: Seq[String] = (1 to 5).map(_ => newStressTopic)
+
+ def newStressTopic: String = s"stress${topicId.getAndIncrement()}"
+
+ private def nextInt(start: Int, end: Int): Int = {
+ start + Random.nextInt(start + end - 1)
+ }
+
+ after {
+ for (topic <- testUtils.getAllTopicsAndPartitionSize().toMap.keys) {
--- End diff --
@zsxwing fix this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66423/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66433/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the issue:
https://github.com/apache/spark/pull/15367
No, if we backport this I would plan to continue to backport changes (that are safe) until the next release. Either way this should not affect what goes into master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15367
Thanks! I'm going to merge this one since the concern from @koeninger is addressed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66432/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/15367
@zsxwing Have you test the maven build?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66432/consoleFull)** for PR 15367 at commit [`0915826`](https://github.com/apache/spark/commit/091582654ba5c4fda8bf6ba39d0248b283f9a6e8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15367
> @zsxwing Have you tested the maven build?
> Test build #66433 has finished for PR 15367 at commit 0915826.
>
> This patch passes all tests.
> This patch merges cleanly.
> This patch adds no public classes.
This one is maven build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/15367
cc @tdas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/15367
@marmbrus @zsxwing I agree its experimental and we should have more flexibility here with backports. I also very much agree that structured streaming in its current state on 2.0 isn't usable - but I'm not super sure that backporting fixes is the best way to do this? Honestly I spend most of my time focused on Python & ML (and I've only really been looking at structured streaming with those two hats on).
I'm really cautious about the idea 2k+ line backport which hasn't even been released otherwise but I don't have any specific objections to the changes its just making me nervous. The fact the whats being backported seems to still be under development is also concerning since doing this backport now puts us in a position of backporting more (not yet merged into mainline) fixes.
Of course - If the people with the most experience in this area all agree (and most of y'all [ @marmbrus @zsxwing @tdas but maybe missing @koeninger ] seem to already be on this PR so I'll leave you to it) that this backport reasonable that is great - it would probably be good to follow up to the original backport mailing list thread and update the wiki as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66423/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66408/consoleFull)** for PR 15367 at commit [`a560190`](https://github.com/apache/spark/commit/a560190dba618698ae2d9728e8b35c5954772467).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL][test-maven]Add Kafka source for Struc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15367
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66432/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15367: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15367
**[Test build #66408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66408/consoleFull)** for PR 15367 at commit [`a560190`](https://github.com/apache/spark/commit/a560190dba618698ae2d9728e8b35c5954772467).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org