You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2019/05/23 04:54:22 UTC

[GitHub] [flink] tweise opened a new pull request #8517: [FLINK-10921] [kinesis] Shard watermark synchronization in Kinesis consumer

tweise opened a new pull request #8517: [FLINK-10921] [kinesis] Shard watermark synchronization in Kinesis consumer
URL: https://github.com/apache/flink/pull/8517
 
 
   ## What is the purpose of the change
   
   * This pull request makes adds support for source synchronization to the Kinesis consumer. Source synchronization aligns shard consumption based on the per-shard watermark.
   
   ## Brief change log
   
   - Adds global watermark aggregation to track the current low watermark across subtasks
   - Adds record emitter to prioritize per-shard queues and align the shard consumers through back pressure
   - Above components are designed to be reusable for the upcoming support in the Kafka consumer.
   - Integration into KinesisDataFetcher 
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   - This is a back port from Lyft internal codebase where it has been tested extensively and at scale.
   - PR includes test for consumer and basic tests for emitter and watermark tracker.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (yes)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes)
     - If yes, how is the feature documented? (user documentation will be added separately)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services