[jira] [Created] (FLINK-23132) flink upgrade issue(1.11.3->1.13.0)

["Jeff Hu (Jira)" <ji...@apache.org>]

Jeff Hu created FLINK-23132:
-------------------------------

             Summary: flink upgrade issue(1.11.3->1.13.0)
                 Key: FLINK-23132
                 URL: https://issues.apache.org/jira/browse/FLINK-23132
             Project: Flink
          Issue Type: Bug
            Reporter: Jeff Hu


 
In order to improve the performance of data process, we store events to a map and do not process them untill event count reaches 100. in the meantime, start a timer in open method, so data is processed every 60 seconds

this works when flink version is *1.11.3*,

after upgrading flink version to *1.13.0*

I found sometimes events were consumed from Kafka continuously, but were not processed in RichFlatMapFunction, it means data was missing. after restarting service, it works well, but several hours later the same thing happened again.

any known issue for this flink version? any suggestions are appreciated.
 {{public class MyJob \{
    public static void main(String[] args) throws Exception {
        ...
        DataStream<String> rawEventSource = env.addSource(flinkKafkaConsumer);
        ...
    }}} {{public class MyMapFunction extends RichFlatMapFunction<String, String> implements Serializable \{

    @Override
    public void open(Configuration parameters) {
        ...
        long periodTimeout = 60;
        pool.scheduleAtFixedRate(() -> {
            // processing data
        }, periodTimeout, periodTimeout, TimeUnit.SECONDS);
    }
    
    @Override
    public void flatMap(String message, Collector<String> out) \{
        // store event to map 
        // count event, 
        // when count = 100, start data processing
    }
}}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)