[jira] [Commented] (FLINK-10225) Cannot access state from a empty taskmanager

["Till Rohrmann (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635530#comment-16635530 ] 

Till Rohrmann commented on FLINK-10225:
---------------------------------------

[~PierreZ], I think we would need to extend the {{KvStateClientProxyImpl}} a bit to make this possible. Instead of having a {{updateKvStateLocationOracle}} method where the {{TaskExecutor}} actively registers specific jobs, we would need to introduce an interface where the {{KvStateClientProxyImpl}} can actively request for a job. Depending on whether the {{TaskExecutor}} knows about the specific job, it can answer it or has to send it to the {{ResourceManager}}.

Before diving into the implementation details, it would be great to properly design the required interface changes. I've also pulled in [~kkl0u] who is the main author of this component.

> Cannot access state from a empty taskmanager
> --------------------------------------------
>
>                 Key: FLINK-10225
>                 URL: https://issues.apache.org/jira/browse/FLINK-10225
>             Project: Flink
>          Issue Type: Bug
>          Components: Queryable State
>    Affects Versions: 1.5.3, 1.6.0
>         Environment: 4tm and 1jm for now on 1.6.0
>            Reporter: Pierre Zemb
>            Priority: Critical
>
> Hi!
> I've started to deploy a small Flink cluster (4tm and 1jm for now on 1.6.0), and deployed a small job on it. Because of the current load, job is completely handled by a single tm. I've created a small proxy that is using [QueryableStateClient|https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/queryablestate/client/QueryableStateClient.html] to access the current state. It is working nicely, except under certain circumstances. It seems to me that I can only access the state through a node that is holding a part of the job. Here's an example:
>  * job on tm1. Pointing QueryableStateClient to tm1. State accessible
>  * job still on tm1. Pointing QueryableStateClient to tm2 (for example). State inaccessible
>  * killing tm1, job is now on tm2. State accessible
>  * job still on tm2. Pointing QueryableStateClient to tm3. State inaccessible
>  * adding some parallelism to spread job on tm1 and tm2. Pointing QueryableStateClient to either tm1 and tm2 is working
>  * job still on tm1 and tm2. Pointing QueryableStateClient to tm3. State inaccessible
> When the state is inaccessible, I can see this (generated [here|https://github.com/apache/flink/blob/release-1.6/flink-queryable-state/flink-queryable-state-runtime/src/main/java/org/apache/flink/queryablestate/client/proxy/KvStateClientProxyHandler.java#L228]):
> {code:java}
> java.lang.RuntimeException: Failed request 0. Caused by: org.apache.flink.queryablestate.exceptions.UnknownLocationException: Could not retrieve location of state=repo-status of job=3ac3bc00b2d5bc0752917186a288d40a. Potential reasons are: i) the state is not ready, or ii) the job does not exist. at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.getKvStateLookupInfo(KvStateClientProxyHandler.java:228) at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.getState(KvStateClientProxyHandler.java:162) at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.executeActionAsync(KvStateClientProxyHandler.java:129) at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.handleRequest(KvStateClientProxyHandler.java:119) at org.apache.flink.queryablestate.client.proxy.KvStateClientProxyHandler.handleRequest(KvStateClientProxyHandler.java:63) at org.apache.flink.queryablestate.network.AbstractServerHandler$AsyncRequestTask.run(AbstractServerHandler.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
> {code}
> Went a bit through the (master branch) code. Class KvStateClientProxy is holding {color:#333333}kvStateLocationOracle the key-value state location oracle for the given JobID. Here's the usage{color}{color:#333333}:{color}
>  * {color:#333333}updateKvStateLocationOracle() in registerQueryableState() (TaskExecutor.java){color}
>  * {color:#333333}registerQueryableState() in associateWithJobManager() (TaskExecutor.java){color}
>  * {color:#333333}associateWithJobManager in establishJobManagerConnection (TaskExecutor.java){color}
>  * {color:#333333}establishJobManagerConnection in jobManagerGainedLeadership (TaskExecutor.java){color}
>  * {color:#333333}jobManagerGainedLeadership in onRegistrationSuccess (JobLeaderService.java){color}
> {color:#333333}It seems that the KvStateLocationOracle map is updated only when the task manager is part of the job. {color}
> {color:#333333}For now, we are creating a List<CompletableFuture<...>> and getting the first CompletableFuture.succeeded future, but that is a workaround.{color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)