[GitHub] [flink] ljurukov opened a new pull request #8111: [FLINK-10195] [RabbitMQ] - RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly

[GitBox <gi...@apache.org>]

ljurukov opened a new pull request #8111: [FLINK-10195] [RabbitMQ] - RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly
URL: https://github.com/apache/flink/pull/8111
 
 
   ## What Does This Change 
   Currently RabbitMQ Connection doesn't backpressure when setting no limit to unacked messages.  Data is stored in an in memory buffer until the heap pops. This resolves this issue by allowing the user to specify a buffer size, limiting the number of data points stored in memory. by default the behavior will be the same.
   
   ## Brief change log
     - Added optional configuration parameter for specifying buffer size
     - Created mechanism for turning on and off the connection as the buffer gets too full or empty
   
   
   ## Verifying this change
   
     - Manually verified the change by running a RabbitMQ instance with 18 million records (~37 GB) of data and and a task manager with a couple of GB or RAM writing slowly to a Sink
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes) - Updated RabbitMQ to use latest
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (JavaDocs)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services