You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/28 02:59:42 UTC
[GitHub] [spark] xuanyuanking commented on a change in pull request #30175: [SPARK-33274][SS] Fix job hang in cp mode when total cores less than total kafka partition
xuanyuanking commented on a change in pull request #30175:
URL: https://github.com/apache/spark/pull/30175#discussion_r565787293
##########
File path: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala
##########
@@ -19,15 +19,24 @@ package org.apache.spark.sql.kafka010
import org.apache.kafka.clients.producer.ProducerRecord
+import org.apache.spark.SparkContext
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.execution.datasources.v2.ContinuousScanExec
import org.apache.spark.sql.execution.streaming.ContinuousTrigger
import org.apache.spark.sql.streaming.Trigger
+import org.apache.spark.sql.test.TestSparkSession
// Run tests in KafkaSourceSuiteBase in continuous execution mode.
class KafkaContinuousSourceSuite extends KafkaSourceSuiteBase with KafkaContinuousTest {
import testImplicits._
+ override protected def createSparkSession = new TestSparkSession(
Review comment:
Why we need to set `spark.executor.cores` in both here and https://github.com/apache/spark/pull/30175/files#diff-d56b5958c6efb7fe729a960641a907b142f0ba0268ec2e31f95e1e42044244e4R43?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochCoordinator.scala
##########
@@ -133,6 +133,13 @@ private[continuous] class EpochCoordinator(
private var currentDriverEpoch = startEpoch
+ val INSTRUCTION_FOR_FEWER_CORES =
Review comment:
Since this message should only be used in `checkTotalCores`, let's move it there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org