Job Manager High Availability using Kubernetes HA Services is not working with custom S3 encrypted client

[Almog Rozencwajg <Al...@niceactimize.com>]

Hi,

We are using Flink version 1.12.2 running on Kubernetes using native Kubernetes integration.
We are trying to use the job manager HA describes in the Flink official documentation Kubernetes HA Services<https://nightlies.apache.org/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html?web=1&wdLOR=c1FF5A075-5403-4C6D-AE70-8F4686D610E9>.
For high-availability.storageDir we are using S3.
We implemented our own S3 client for the Hadoop plugin in order to use S3 client side encryption.
The job manager failed when the Hadoop trying to upload the recovery metadata to S3 using the multipart upload API.
The Hadoop uploads the data in parts calling the upload part method and failed on:
'com.amazonaws.SdkClientException: Invalid part size: part sizes for encrypted multipart uploads must be multiples of the cipher block size (16) with the exception of the last part'.
When using S3 client side encryption with multipart upload API there is a limitation on the part size.
Is there a way we can control via configuration the part size of the recovery data uploaded via Flink HA?

Thanks,
Almog

Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.