[GitHub] StefanRRichter opened a new pull request #6994: [FLINK-10720][tests] Add deployment end-to-end stress test with many …

[GitBox <gi...@apache.org>]

StefanRRichter opened a new pull request #6994: [FLINK-10720][tests] Add deployment end-to-end stress test with many …
URL: https://github.com/apache/flink/pull/6994
 
 
   …inflated task deployment desciptors
   
   ## What is the purpose of the change
   
   This PR provides a nighly end-to-end test to simulate a heavy deployment, many task with large task deployment desciptors. This is a stress test for the deployment process that create, serializes, and sends the deployment descriptors.
   
   The idea to create inflated deployment descriptors in the job is to use many union operator states with many partitions. They will be replicated in the deployment to all tasks. For example, if we have 100 union states, each with 50 partitions, and a parallelism of 256, the meta data for each partition will be at least the long that represents the partition offset. So the estimated deployment in a recovery amounts to more than 100 (states) x 50 (partitions) x 256 x 256 (parallelism squared) x 8 (size of long) ~ 2,4GB.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services