How to do code isolation if muiltple jobs run on the same taskmanager process?

[Lei Wang <le...@gmail.com>]

Consider the following situation.

Two jobs builed in the same jar, they will share some common code, for
example, some static constants variables.
Currently they are running on the same task manager process.

I  killed job1, changed the static variable to another and then submit it
again.
Does another job will get the new value of the static variable or still use
the old value?

Thanks,
Lei

Re: How to do code isolation if muiltple jobs run on the same taskmanager process?

[Arvid Heise <ar...@apache.org>]

Hi Lei,

yes, metaspace would run out eventually if you run too much in parallel.
All finished jobs will close the classloaders and free the metaspace memory.

For newer setups, we recommend creating an ad-hoc cluster for each Flink
application for this and several other reasons. If you are already using
K8s, I'd definitively switch to per-application clusters.

You could also have a look at the Ververica community editions for easy
managing of such applications. [1]

[1] https://www.ververica.com/getting-started

On Fri, Mar 12, 2021 at 4:15 AM Lei Wang <le...@gmail.com> wrote:

> Thanks Arvid,
>
> If too many jobs run in the same task manager JVM,  will it  cause too
> much metaspace memory occupation?
>
>
> Thanks,
> Lei
>
> On Thu, Mar 11, 2021 at 11:54 PM Arvid Heise <ar...@apache.org> wrote:
>
>> Hi Lei,
>>
>> each application has its own classloader as such each static constant
>> exists multiple times (1 per job). So there should be no interference. You
>> could verify it by logging the value of the constant and see it yourself.
>>
>> Best,
>>
>> Arvid
>>
>> On Thu, Mar 11, 2021 at 7:11 AM Lei Wang <le...@gmail.com> wrote:
>>
>>> Consider the following situation.
>>>
>>> Two jobs builed in the same jar, they will share some common code, for
>>> example, some static constants variables.
>>> Currently they are running on the same task manager process.
>>>
>>> I  killed job1, changed the static variable to another and then submit
>>> it again.
>>> Does another job will get the new value of the static variable or still
>>> use the old value?
>>>
>>> Thanks,
>>> Lei
>>>
>>>
>>>

Re: How to do code isolation if muiltple jobs run on the same taskmanager process?

[Lei Wang <le...@gmail.com>]

Thanks Arvid,

If too many jobs run in the same task manager JVM,  will it  cause too much
metaspace memory occupation?


Thanks,
Lei

On Thu, Mar 11, 2021 at 11:54 PM Arvid Heise <ar...@apache.org> wrote:

> Hi Lei,
>
> each application has its own classloader as such each static constant
> exists multiple times (1 per job). So there should be no interference. You
> could verify it by logging the value of the constant and see it yourself.
>
> Best,
>
> Arvid
>
> On Thu, Mar 11, 2021 at 7:11 AM Lei Wang <le...@gmail.com> wrote:
>
>> Consider the following situation.
>>
>> Two jobs builed in the same jar, they will share some common code, for
>> example, some static constants variables.
>> Currently they are running on the same task manager process.
>>
>> I  killed job1, changed the static variable to another and then submit it
>> again.
>> Does another job will get the new value of the static variable or still
>> use the old value?
>>
>> Thanks,
>> Lei
>>
>>
>>

Re: How to do code isolation if muiltple jobs run on the same taskmanager process?

[Arvid Heise <ar...@apache.org>]

Hi Lei,

each application has its own classloader as such each static constant
exists multiple times (1 per job). So there should be no interference. You
could verify it by logging the value of the constant and see it yourself.

Best,

Arvid

On Thu, Mar 11, 2021 at 7:11 AM Lei Wang <le...@gmail.com> wrote:

> Consider the following situation.
>
> Two jobs builed in the same jar, they will share some common code, for
> example, some static constants variables.
> Currently they are running on the same task manager process.
>
> I  killed job1, changed the static variable to another and then submit it
> again.
> Does another job will get the new value of the static variable or still
> use the old value?
>
> Thanks,
> Lei
>
>
>