Uploading jar on multiple flink job managers

[Puneet Duggal <pu...@gmail.com>]

Hi,

So currently we are using flink 1.12 in HA mode on production. There are 3 job managers (1 leader and 2 standby). When I am uploading a jar on one of the job managers, somehow it is not reflected on other job managers. Is there any way where I can achieve a behaviour where uploading jar to a single job manager also gets reflected in other job managers in HA as well?

The problem that I am facing due to this is that when the jar is uploaded on let's say 'A' job manager, but when sending a job submit a request using uploaded jar on 'B' job manager, I get an error saying jar not found.

Thanks, 
Puneet

Re: Uploading jar on multiple flink job managers

[Timo Walther <tw...@apache.org>]

Hi Puneet,

are we talking about the `web.upload.dir` [1] ? Maybe others have a 
better solution for your problem, but have you thought about configuring 
an NFS or some other distributed file system as the JAR directory? In 
this case it should be available to all JobManagers.

Regards,
Timo

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#web-upload-dir

On 25.12.21 08:59, Puneet Duggal wrote:
> Hi,
> 
> So currently we are using flink 1.12 in HA mode on production. There are 
> 3 job managers (1 leader and 2 standby). When I am uploading a jar on 
> one of the job managers, somehow it is not reflected on other job 
> managers. Is there any way where I can achieve a behaviour where 
> uploading jar to a single job manager also gets reflected in other job 
> managers in HA as well?
> 
> The problem that I am facing due to this is that when the jar is 
> uploaded on let's say 'A' job manager, but when sending a job submit a 
> request using uploaded jar on 'B' job manager, I get an error saying jar 
> not found.
> 
> Thanks,
> Puneet