You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/20 05:59:56 UTC

[GitHub] [spark] HeartSaVioR opened a new pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

HeartSaVioR opened a new pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960
 
 
   ### What changes were proposed in this pull request?
   
   This patch upgrades the version of Kafka to 2.4, which supports Scala 2.13.
   
   There're some incompatible changes in Kafka 2.4 which the patch addresses as well:
   
   * `ZkUtils` is removed -> Replaced with `KafkaZkClient`
   * Majority of methods are removed in `AdminUtils` -> Replaced with `AdminZkClient`
   * Method signature of `Scheduler.schedule` is changed (return type) -> leverage `DeterministicScheduler` to avoid implementing `ScheduledFuture`
   
   ### Why are the changes needed?
   
   * Kafka 2.4 supports Scala 2.13
   * Kafka 2.4 is prerequisite to resolve [SPARK-28367](https://issues.apache.org/jira/browse/SPARK-28367), as it requires KIP-396
   
   ### Does this PR introduce any user-facing change?
   
   No, as Kafka API is known to be compatible across versions.
   
   ### How was this patch tested?
   
   Existing UTs

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567978839
 
 
   **[Test build #115627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115627/testReport)** for PR 26960 at commit [`4cca9d2`](https://github.com/apache/spark/commit/4cca9d27f08f595c1e1dd8df6bdbf8024ec6e59d).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-571983511
 
 
   Aaaand of course same best wishes to everybody :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567927191
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115618/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567927184
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567952124
 
 
   **[Test build #115626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115626/testReport)** for PR 26960 at commit [`571d3d4`](https://github.com/apache/spark/commit/571d3d4f236f268bee9433bb4ae7c37fc826f7c4).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-568041990
 
 
   **[Test build #115627 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115627/testReport)** for PR 26960 at commit [`4cca9d2`](https://github.com/apache/spark/commit/4cca9d27f08f595c1e1dd8df6bdbf8024ec6e59d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360386582
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala
 ##########
 @@ -41,57 +42,33 @@ import org.apache.kafka.common.utils.Time
  */
 private[kafka010] class MockScheduler(val time: Time) extends Scheduler {
 
-  /* a priority queue of tasks ordered by next execution time */
-  var tasks = new PriorityQueue[MockTask]()
+  val scheduler = new DeterministicScheduler()
 
   def isStarted: Boolean = true
 
   def startup(): Unit = {}
 
   def shutdown(): Unit = synchronized {
-    tasks.foreach(_.fun())
-    tasks.clear()
+    scheduler.runUntilIdle()
   }
 
-  /**
-   * Check for any tasks that need to execute. Since this is a mock scheduler this check only occurs
-   * when this method is called and the execution happens synchronously in the calling thread.
-   * If you are using the scheduler associated with a MockTime instance this call
-   * will be triggered automatically.
-   */
-  def tick(): Unit = synchronized {
-    val now = time.milliseconds
-    while(!tasks.isEmpty && tasks.head.nextExecution <= now) {
-      /* pop and execute the task with the lowest next execution time */
-      val curr = tasks.dequeue
-      curr.fun()
-      /* if the task is periodic, reschedule it and re-enqueue */
-      if(curr.periodic) {
-        curr.nextExecution += curr.period
-        this.tasks += curr
-      }
-    }
+  def tick(duration: Long, timeUnit: TimeUnit): Unit = synchronized {
+    scheduler.tick(duration, timeUnit)
   }
 
   def schedule(
       name: String,
       fun: () => Unit,
       delay: Long = 0,
       period: Long = -1,
-      unit: TimeUnit = TimeUnit.MILLISECONDS): Unit = synchronized {
-    tasks += MockTask(name, fun, time.milliseconds + delay, period = period)
-    tick()
-  }
-
-}
-
-case class MockTask(
-    val name: String,
-    val fun: () => Unit,
-    var nextExecution: Long,
-    val period: Long) extends Ordered[MockTask] {
-  def periodic: Boolean = period >= 0
-  def compare(t: MockTask): Int = {
-    java.lang.Long.compare(t.nextExecution, nextExecution)
+      unit: TimeUnit = TimeUnit.MILLISECONDS): ScheduledFuture[_] = synchronized {
+    val runnable = new Runnable {
+      override def run(): Unit = fun()
+    }
+    if(period >= 0) {
+      scheduler.scheduleAtFixedRate(runnable, delay, period, unit)
 
 Review comment:
   It doesn't allow to just pass `fun`, but allow `() => fun()` so will use it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567955652
 
 
   **[Test build #115626 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115626/testReport)** for PR 26960 at commit [`571d3d4`](https://github.com/apache/spark/commit/571d3d4f236f268bee9433bb4ae7c37fc826f7c4).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-572001777
 
 
   Oh .. I see.. Happy to see you back :D.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-568042991
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115627/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567978839
 
 
   **[Test build #115627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115627/testReport)** for PR 26960 at commit [`4cca9d2`](https://github.com/apache/spark/commit/4cca9d27f08f595c1e1dd8df6bdbf8024ec6e59d).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360661053
 
 

 ##########
 File path: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
 ##########
 @@ -626,9 +624,9 @@ class KafkaTestUtils(
     def isPropagated = server.dataPlaneRequestProcessor.metadataCache
         .getPartitionInfo(topic, partition) match {
       case Some(partitionState) =>
-        zkUtils.getLeaderForPartition(topic, partition).isDefined &&
-          Request.isValidBrokerId(partitionState.basePartitionState.leader) &&
-          !partitionState.basePartitionState.replicas.isEmpty
+        zkClient.getLeaderForPartition(new TopicPartition(topic, partition)).isDefined &&
+          Request.isValidBrokerId(partitionState.leader) &&
 
 Review comment:
   Got it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567955886
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115626/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360661947
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
 ##########
 @@ -27,13 +27,14 @@ import scala.annotation.tailrec
 import scala.collection.JavaConverters._
 import scala.util.control.NonFatal
 
-import kafka.admin.AdminUtils
 import kafka.api.Request
 import kafka.server.{KafkaConfig, KafkaServer}
-import kafka.utils.ZkUtils
+import kafka.zk.{AdminZkClient, KafkaZkClient}
 import org.apache.kafka.clients.producer.{KafkaProducer, ProducerRecord}
+import org.apache.kafka.common.TopicPartition
 import org.apache.kafka.common.network.ListenerName
 import org.apache.kafka.common.serialization.StringSerializer
+import org.apache.kafka.common.utils.SystemTime
 
 Review comment:
   `import org.apache.kafka.common.utils.Time`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567955875
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567881131
 
 
   retest this, please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567884707
 
 
   **[Test build #115618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115618/testReport)** for PR 26960 at commit [`ad03fca`](https://github.com/apache/spark/commit/ad03fca2069eeba1545dea2ef8058397e034a83a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360366350
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala
 ##########
 @@ -41,57 +42,33 @@ import org.apache.kafka.common.utils.Time
  */
 private[kafka010] class MockScheduler(val time: Time) extends Scheduler {
 
-  /* a priority queue of tasks ordered by next execution time */
-  var tasks = new PriorityQueue[MockTask]()
+  val scheduler = new DeterministicScheduler()
 
   def isStarted: Boolean = true
 
   def startup(): Unit = {}
 
   def shutdown(): Unit = synchronized {
-    tasks.foreach(_.fun())
-    tasks.clear()
+    scheduler.runUntilIdle()
   }
 
-  /**
-   * Check for any tasks that need to execute. Since this is a mock scheduler this check only occurs
-   * when this method is called and the execution happens synchronously in the calling thread.
-   * If you are using the scheduler associated with a MockTime instance this call
-   * will be triggered automatically.
-   */
-  def tick(): Unit = synchronized {
-    val now = time.milliseconds
-    while(!tasks.isEmpty && tasks.head.nextExecution <= now) {
-      /* pop and execute the task with the lowest next execution time */
-      val curr = tasks.dequeue
-      curr.fun()
-      /* if the task is periodic, reschedule it and re-enqueue */
-      if(curr.periodic) {
-        curr.nextExecution += curr.period
-        this.tasks += curr
-      }
-    }
+  def tick(duration: Long, timeUnit: TimeUnit): Unit = synchronized {
+    scheduler.tick(duration, timeUnit)
   }
 
   def schedule(
       name: String,
       fun: () => Unit,
       delay: Long = 0,
       period: Long = -1,
-      unit: TimeUnit = TimeUnit.MILLISECONDS): Unit = synchronized {
-    tasks += MockTask(name, fun, time.milliseconds + delay, period = period)
-    tick()
-  }
-
-}
-
-case class MockTask(
-    val name: String,
-    val fun: () => Unit,
-    var nextExecution: Long,
-    val period: Long) extends Ordered[MockTask] {
-  def periodic: Boolean = period >= 0
-  def compare(t: MockTask): Int = {
-    java.lang.Long.compare(t.nextExecution, nextExecution)
+      unit: TimeUnit = TimeUnit.MILLISECONDS): ScheduledFuture[_] = synchronized {
+    val runnable = new Runnable {
+      override def run(): Unit = fun()
+    }
+    if(period >= 0) {
 
 Review comment:
   Nit: space after if

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567937999
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567888783
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360661930
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
 ##########
 @@ -85,19 +87,27 @@ private[kafka010] class KafkaTestUtils extends Logging {
     s"$brokerHost:$brokerPort"
   }
 
-  def zookeeperClient: ZkUtils = {
+  def zookeeperClient: KafkaZkClient = {
     assert(zkReady, "Zookeeper not setup yet or already torn down, cannot get zookeeper client")
-    Option(zkUtils).getOrElse(
+    Option(zkClient).getOrElse(
       throw new IllegalStateException("Zookeeper client is not yet initialized"))
   }
 
+  def adminClient: AdminZkClient = {
+    assert(zkReady, "Zookeeper not setup yet or already torn down, cannot get zookeeper client")
+    Option(admClient).getOrElse(
+      throw new IllegalStateException("Admin client is not yet initialized"))
+  }
+
   // Set up the Embedded Zookeeper server and get the proper Zookeeper port
   private def setupEmbeddedZookeeper(): Unit = {
     // Zookeeper server startup
     zookeeper = new EmbeddedZookeeper(s"$zkHost:$zkPort")
     // Get the actual zookeeper binding port
     zkPort = zookeeper.actualPort
-    zkUtils = ZkUtils(s"$zkHost:$zkPort", zkSessionTimeout, zkConnectionTimeout, false)
+    zkClient = KafkaZkClient(s"$zkHost:$zkPort", isSecure = false, zkSessionTimeout,
+      zkConnectionTimeout, 1, new SystemTime())
 
 Review comment:
   `new SystemTime()` -> `Time.SYSTEM`?
   (To avoid a conflict with `import org.apache.spark.streaming.Time`, this should be `KTime.SYSTEM` if we use this.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567938007
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20424/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567888783
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567925556
 
 
   **[Test build #115618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115618/testReport)** for PR 26960 at commit [`ad03fca`](https://github.com/apache/spark/commit/ad03fca2069eeba1545dea2ef8058397e034a83a).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-568042982
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-571970009
 
 
   Happy New Year @dongjoon-hyun ! Thanks for your warm welcome :)
   After a relatively long pause hope I can continue to help the community...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567955875
 
 
   Merged build finished. Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-568199859
 
 
   Thank you so much, @HeartSaVioR !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-571981374
 
 
   Everybody went for holidays :-)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567937999
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567927184
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567938007
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20424/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-571739443
 
 
   Finally, you are back, @gaborgsomogyi . :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360662563
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala
 ##########
 @@ -41,57 +42,33 @@ import org.apache.kafka.common.utils.Time
  */
 private[kafka010] class MockScheduler(val time: Time) extends Scheduler {
 
-  /* a priority queue of tasks ordered by next execution time */
-  var tasks = new PriorityQueue[MockTask]()
+  val scheduler = new DeterministicScheduler()
 
   def isStarted: Boolean = true
 
   def startup(): Unit = {}
 
   def shutdown(): Unit = synchronized {
-    tasks.foreach(_.fun())
-    tasks.clear()
+    scheduler.runUntilIdle()
   }
 
-  /**
-   * Check for any tasks that need to execute. Since this is a mock scheduler this check only occurs
-   * when this method is called and the execution happens synchronously in the calling thread.
-   * If you are using the scheduler associated with a MockTime instance this call
-   * will be triggered automatically.
-   */
-  def tick(): Unit = synchronized {
-    val now = time.milliseconds
-    while(!tasks.isEmpty && tasks.head.nextExecution <= now) {
-      /* pop and execute the task with the lowest next execution time */
-      val curr = tasks.dequeue
-      curr.fun()
-      /* if the task is periodic, reschedule it and re-enqueue */
-      if(curr.periodic) {
-        curr.nextExecution += curr.period
-        this.tasks += curr
-      }
-    }
+  def tick(duration: Long, timeUnit: TimeUnit): Unit = synchronized {
+    scheduler.tick(duration, timeUnit)
   }
 
   def schedule(
       name: String,
       fun: () => Unit,
       delay: Long = 0,
       period: Long = -1,
-      unit: TimeUnit = TimeUnit.MILLISECONDS): Unit = synchronized {
-    tasks += MockTask(name, fun, time.milliseconds + delay, period = period)
-    tick()
-  }
-
-}
-
-case class MockTask(
 
 Review comment:
   Thank you for removing this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-571502110
 
 
   Thanks guys to take care.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567927191
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115618/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567884707
 
 
   **[Test build #115618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115618/testReport)** for PR 26960 at commit [`ad03fca`](https://github.com/apache/spark/commit/ad03fca2069eeba1545dea2ef8058397e034a83a).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567980873
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20426/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567955886
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115626/
   Test FAILed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567980868
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-568265920
 
 
   I just addressed these review comments in #26979 as I'm seeing benefit on the change. Please take a look. Thanks in advance!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567952124
 
 
   **[Test build #115626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/115626/testReport)** for PR 26960 at commit [`571d3d4`](https://github.com/apache/spark/commit/571d3d4f236f268bee9433bb4ae7c37fc826f7c4).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360661930
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
 ##########
 @@ -85,19 +87,27 @@ private[kafka010] class KafkaTestUtils extends Logging {
     s"$brokerHost:$brokerPort"
   }
 
-  def zookeeperClient: ZkUtils = {
+  def zookeeperClient: KafkaZkClient = {
     assert(zkReady, "Zookeeper not setup yet or already torn down, cannot get zookeeper client")
-    Option(zkUtils).getOrElse(
+    Option(zkClient).getOrElse(
       throw new IllegalStateException("Zookeeper client is not yet initialized"))
   }
 
+  def adminClient: AdminZkClient = {
+    assert(zkReady, "Zookeeper not setup yet or already torn down, cannot get zookeeper client")
+    Option(admClient).getOrElse(
+      throw new IllegalStateException("Admin client is not yet initialized"))
+  }
+
   // Set up the Embedded Zookeeper server and get the proper Zookeeper port
   private def setupEmbeddedZookeeper(): Unit = {
     // Zookeeper server startup
     zookeeper = new EmbeddedZookeeper(s"$zkHost:$zkPort")
     // Get the actual zookeeper binding port
     zkPort = zookeeper.actualPort
-    zkUtils = ZkUtils(s"$zkHost:$zkPort", zkSessionTimeout, zkConnectionTimeout, false)
+    zkClient = KafkaZkClient(s"$zkHost:$zkPort", isSecure = false, zkSessionTimeout,
+      zkConnectionTimeout, 1, new SystemTime())
 
 Review comment:
   `new SystemTime()` -> `Time.SYSTEM`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-568042982
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360419996
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala
 ##########
 @@ -41,57 +42,33 @@ import org.apache.kafka.common.utils.Time
  */
 private[kafka010] class MockScheduler(val time: Time) extends Scheduler {
 
-  /* a priority queue of tasks ordered by next execution time */
-  var tasks = new PriorityQueue[MockTask]()
+  val scheduler = new DeterministicScheduler()
 
   def isStarted: Boolean = true
 
   def startup(): Unit = {}
 
   def shutdown(): Unit = synchronized {
-    tasks.foreach(_.fun())
-    tasks.clear()
+    scheduler.runUntilIdle()
   }
 
-  /**
-   * Check for any tasks that need to execute. Since this is a mock scheduler this check only occurs
-   * when this method is called and the execution happens synchronously in the calling thread.
-   * If you are using the scheduler associated with a MockTime instance this call
-   * will be triggered automatically.
-   */
-  def tick(): Unit = synchronized {
-    val now = time.milliseconds
-    while(!tasks.isEmpty && tasks.head.nextExecution <= now) {
-      /* pop and execute the task with the lowest next execution time */
-      val curr = tasks.dequeue
-      curr.fun()
-      /* if the task is periodic, reschedule it and re-enqueue */
-      if(curr.periodic) {
-        curr.nextExecution += curr.period
-        this.tasks += curr
-      }
-    }
+  def tick(duration: Long, timeUnit: TimeUnit): Unit = synchronized {
+    scheduler.tick(duration, timeUnit)
   }
 
   def schedule(
       name: String,
       fun: () => Unit,
       delay: Long = 0,
       period: Long = -1,
-      unit: TimeUnit = TimeUnit.MILLISECONDS): Unit = synchronized {
-    tasks += MockTask(name, fun, time.milliseconds + delay, period = period)
-    tick()
-  }
-
-}
-
-case class MockTask(
-    val name: String,
-    val fun: () => Unit,
-    var nextExecution: Long,
-    val period: Long) extends Ordered[MockTask] {
-  def periodic: Boolean = period >= 0
-  def compare(t: MockTask): Int = {
-    java.lang.Long.compare(t.nextExecution, nextExecution)
+      unit: TimeUnit = TimeUnit.MILLISECONDS): ScheduledFuture[_] = synchronized {
+    val runnable = new Runnable {
+      override def run(): Unit = fun()
+    }
+    if(period >= 0) {
+      scheduler.scheduleAtFixedRate(runnable, delay, period, unit)
 
 Review comment:
   `schedule` has overloaded methods which reject `() => fun()`. Just reverted back.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567980873
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20426/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-568042991
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115627/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-568263367
 
 
   Thanks all for reviewing and merging!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
gaborgsomogyi commented on issue #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960#issuecomment-571982804
 
 
   My "holiday" was bit forced because of my health but hope now it will be better...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360386582
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala
 ##########
 @@ -41,57 +42,33 @@ import org.apache.kafka.common.utils.Time
  */
 private[kafka010] class MockScheduler(val time: Time) extends Scheduler {
 
-  /* a priority queue of tasks ordered by next execution time */
-  var tasks = new PriorityQueue[MockTask]()
+  val scheduler = new DeterministicScheduler()
 
   def isStarted: Boolean = true
 
   def startup(): Unit = {}
 
   def shutdown(): Unit = synchronized {
-    tasks.foreach(_.fun())
-    tasks.clear()
+    scheduler.runUntilIdle()
   }
 
-  /**
-   * Check for any tasks that need to execute. Since this is a mock scheduler this check only occurs
-   * when this method is called and the execution happens synchronously in the calling thread.
-   * If you are using the scheduler associated with a MockTime instance this call
-   * will be triggered automatically.
-   */
-  def tick(): Unit = synchronized {
-    val now = time.milliseconds
-    while(!tasks.isEmpty && tasks.head.nextExecution <= now) {
-      /* pop and execute the task with the lowest next execution time */
-      val curr = tasks.dequeue
-      curr.fun()
-      /* if the task is periodic, reschedule it and re-enqueue */
-      if(curr.periodic) {
-        curr.nextExecution += curr.period
-        this.tasks += curr
-      }
-    }
+  def tick(duration: Long, timeUnit: TimeUnit): Unit = synchronized {
+    scheduler.tick(duration, timeUnit)
   }
 
   def schedule(
       name: String,
       fun: () => Unit,
       delay: Long = 0,
       period: Long = -1,
-      unit: TimeUnit = TimeUnit.MILLISECONDS): Unit = synchronized {
-    tasks += MockTask(name, fun, time.milliseconds + delay, period = period)
-    tick()
-  }
-
-}
-
-case class MockTask(
-    val name: String,
-    val fun: () => Unit,
-    var nextExecution: Long,
-    val period: Long) extends Ordered[MockTask] {
-  def periodic: Boolean = period >= 0
-  def compare(t: MockTask): Int = {
-    java.lang.Long.compare(t.nextExecution, nextExecution)
+      unit: TimeUnit = TimeUnit.MILLISECONDS): ScheduledFuture[_] = synchronized {
+    val runnable = new Runnable {
+      override def run(): Unit = fun()
+    }
+    if(period >= 0) {
+      scheduler.scheduleAtFixedRate(runnable, delay, period, unit)
 
 Review comment:
   It doesn't allow to just pass `fun`, but allows `() => fun()` so will use it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567888799
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20417/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade Kafka to 2.4.0
URL: https://github.com/apache/spark/pull/26960
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] srowen commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360366576
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/mocks/MockScheduler.scala
 ##########
 @@ -41,57 +42,33 @@ import org.apache.kafka.common.utils.Time
  */
 private[kafka010] class MockScheduler(val time: Time) extends Scheduler {
 
-  /* a priority queue of tasks ordered by next execution time */
-  var tasks = new PriorityQueue[MockTask]()
+  val scheduler = new DeterministicScheduler()
 
   def isStarted: Boolean = true
 
   def startup(): Unit = {}
 
   def shutdown(): Unit = synchronized {
-    tasks.foreach(_.fun())
-    tasks.clear()
+    scheduler.runUntilIdle()
   }
 
-  /**
-   * Check for any tasks that need to execute. Since this is a mock scheduler this check only occurs
-   * when this method is called and the execution happens synchronously in the calling thread.
-   * If you are using the scheduler associated with a MockTime instance this call
-   * will be triggered automatically.
-   */
-  def tick(): Unit = synchronized {
-    val now = time.milliseconds
-    while(!tasks.isEmpty && tasks.head.nextExecution <= now) {
-      /* pop and execute the task with the lowest next execution time */
-      val curr = tasks.dequeue
-      curr.fun()
-      /* if the task is periodic, reschedule it and re-enqueue */
-      if(curr.periodic) {
-        curr.nextExecution += curr.period
-        this.tasks += curr
-      }
-    }
+  def tick(duration: Long, timeUnit: TimeUnit): Unit = synchronized {
+    scheduler.tick(duration, timeUnit)
   }
 
   def schedule(
       name: String,
       fun: () => Unit,
       delay: Long = 0,
       period: Long = -1,
-      unit: TimeUnit = TimeUnit.MILLISECONDS): Unit = synchronized {
-    tasks += MockTask(name, fun, time.milliseconds + delay, period = period)
-    tick()
-  }
-
-}
-
-case class MockTask(
-    val name: String,
-    val fun: () => Unit,
-    var nextExecution: Long,
-    val period: Long) extends Ordered[MockTask] {
-  def periodic: Boolean = period >= 0
-  def compare(t: MockTask): Int = {
-    java.lang.Long.compare(t.nextExecution, nextExecution)
+      unit: TimeUnit = TimeUnit.MILLISECONDS): ScheduledFuture[_] = synchronized {
+    val runnable = new Runnable {
+      override def run(): Unit = fun()
+    }
+    if(period >= 0) {
+      scheduler.scheduleAtFixedRate(runnable, delay, period, unit)
 
 Review comment:
   If it takes a Runnable, can you just pass `fun`? If the SAM rules allow it it should pass it as an implementation of Runnable. no big deal of course

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567938505
 
 
   > What implication does this have for compatibility with Kafka brokers?
   
   Kafka community hasn't been mentioning compatibility with Kafka brokers across versions, as they have broad backward compatibility if I understand correctly. They're versioning the protocols.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567888799
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20417/
   Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #26960: [SPARK-28144][SPARK-29294][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#discussion_r360661947
 
 

 ##########
 File path: external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
 ##########
 @@ -27,13 +27,14 @@ import scala.annotation.tailrec
 import scala.collection.JavaConverters._
 import scala.util.control.NonFatal
 
-import kafka.admin.AdminUtils
 import kafka.api.Request
 import kafka.server.{KafkaConfig, KafkaServer}
-import kafka.utils.ZkUtils
+import kafka.zk.{AdminZkClient, KafkaZkClient}
 import org.apache.kafka.clients.producer.{KafkaProducer, ProducerRecord}
+import org.apache.kafka.common.TopicPartition
 import org.apache.kafka.common.network.ListenerName
 import org.apache.kafka.common.serialization.StringSerializer
+import org.apache.kafka.common.utils.SystemTime
 
 Review comment:
   `import org.apache.kafka.common.utils.{Time => KTime}`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #26960: [SPARK-28144][SPARK-29294][SQL][SS] Upgrade the version of Kafka to 2.4
URL: https://github.com/apache/spark/pull/26960#issuecomment-567980868
 
 
   Merged build finished. Test PASSed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org