You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2019/12/12 02:13:00 UTC
[jira] [Resolved] (SPARK-30222) Still getting KafkaConsumer cache
hitting max capacity of 64, removing consumer for CacheKe
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-30222.
----------------------------------
Fix Version/s: (was: 2.4.1)
Target Version/s: (was: 2.4.1)
Resolution: Invalid
> Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-30222
> URL: https://issues.apache.org/jira/browse/SPARK-30222
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.1
> Environment: {{Below are the logs.}}
> 2019-12-11 08:33:31,504 [Executor task launch worker for task 1050] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-21)
> 2019-12-11 08:33:32,493 [Executor task launch worker for task 1051] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-5)
> 2019-12-11 08:33:32,570 [Executor task launch worker for task 1052] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-9)
> 2019-12-11 08:33:33,441 [Executor task launch worker for task 1053] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-17)
> 2019-12-11 08:33:33,619 [Executor task launch worker for task 1054] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-2)
> 2019-12-11 08:33:34,474 [Executor task launch worker for task 1055] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-10)
> 2019-12-11 08:33:35,006 [Executor task launch worker for task 1056] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-6)
> 2019-12-11 08:33:36,326 [Executor task launch worker for task 1057] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-14)
> 2019-12-11 08:33:36,634 [Executor task launch worker for task 1058] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-0)
> 2019-12-11 08:33:37,496 [Executor task launch worker for task 1059] WARN org.apache.spark.sql.kafka010.KafkaDataConsumer - KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKey(spark-kafka-source-93ee3689-79f9-42e8-b1ee-e856570205ae-1923743483-executor,COMPANY_TRANSACTIONS_INBOUND-19)
> 2019-12-11 08:33:39,183 [stream execution thread for [id = b3aec196-e4f2-4ef9-973b-d5685eba917e, runId = 5c35a63a-16ad-4899-b732-1019397770bd]] WARN org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor - Current batch is falling behind. The trigger interval is 15000 milliseconds, but spent 63438 milliseconds
> Reporter: Shyam
> Priority: Major
>
> Me using spark-sql-2.4.1 version with Kafka 0.10 v.
> While I try to consume data by consumer. it gives error below even after setting
> .option("spark.sql.kafkaConsumerCache.capacity",128)
>
> {{Dataset<Row> df = sparkSession}}
> {{ .readStream()}}
> {{ .format("kafka")}}
> {{ .option("kafka.bootstrap.servers", SERVERS)}}
> {{ .option("subscribe", TOPIC) }}{{}}
> {{ .option("spark.sql.kafkaConsumerCache.capacity",128) }}
> {{ }}
> {{ .load();}}
> {{}}
> {{}}
> {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org