You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gabor Somogyi (JIRA)" <ji...@apache.org> on 2018/12/12 08:47:01 UTC

[jira] [Closed] (SPARK-22606) There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access"

     [ https://issues.apache.org/jira/browse/SPARK-22606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabor Somogyi closed SPARK-22606.
---------------------------------

> There may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access" 
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22606
>                 URL: https://issues.apache.org/jira/browse/SPARK-22606
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 2.2.0
>            Reporter: eaton
>            Priority: Minor
>             Fix For: 2.4.0
>
>
> If the value of param 'spark.streaming.concurrentJobs' is more than one, and the value of param 'spark.executor.cores' is more than one, there may be two or more tasks in one executor will use the same kafka consumer at the same time, then it will throw an exception: "KafkaConsumer is not safe for multi-threaded access" 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org