[jira] [Created] (FLINK-14523) Flink Stops Consuming from Kafka after Leader Election

["James B. Fitzgerald (Jira)" <ji...@apache.org>]

James B. Fitzgerald created FLINK-14523:
-------------------------------------------

             Summary: Flink Stops Consuming from Kafka after Leader Election
                 Key: FLINK-14523
                 URL: https://issues.apache.org/jira/browse/FLINK-14523
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka
         Environment: In AWS we run the app on EMR with the following versions:


 * EMR Release - emr-5.15.0
 * Hadoop distribution - Amazon 2.8.3
 * Flink - Flink 1.4.2

We submit the job to the cluster as an EMR step using ***command-runner.jar*. We submit the job with the following arguments:

{code:java}
"Args": [
  "flink", "run", "-m", "yarn-cluster", 
  "-c", "com.salesforce.sde.streamingsearches.StreamingSearchesJob", 
  "-yst", "-ys", "4", "-yn", "10", "-yjm", "2800", "-ytm", "2800",
  "-ynm", "streaming-searches-prod",
  "-d", "/home/hadoop/streaming-searches-1.0-SNAPSHOT.jar"        
]{code}
Additionally we build our application jar with Flink 1.4.2 and Kafka 0.11.

 
            Reporter: James B. Fitzgerald


We have a Flink application running in AWS on EMR that streams input from a single Kafka topic. Whenever there is a Kafka leader election for any partition of the input topic, our Flink application stops consuming from Kafka entirely. To begin consuming from Kafka again the YARN app must be killed and restarted. We run this same application on premises and in AWS on EMR. We have only observed this behavior when it is running on EMR. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)