You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Juha Mynttinen <ju...@aiven.io> on 2021/08/24 10:05:31 UTC
web.timeout usage in the code
Hey,
I was looking at the web.timeout configuration option described as "Timeout
for asynchronous operations by the web monitor in milliseconds". I'm
interested in where and how it's used internally.
I've understood it's the timeout when the Web UI calls something using Rpc
(Akka). Correct? Unfortunately, I can't spot where the parameter is
actually used so that it would have any effect. Maybe I've missed something?
The param is defined in WebOptions and used in RestHandlerConfiguration.
The param is stored in LeaderRetrievalHandler [1]. That class merely acts
as a holder for the param and e.g. the REST handlers use it. If we look at
one of the handlers, e.g. ClusterOverviewHandler, in the end it
calls requestClusterOverview in Dispatcher [2]. That has two usages of the
timeOut, but if you follow them all the way, the param is unused. I've
checked multiple handlers and I couldn't spot any that would really use the
param for something.
Retrieving the leader (that is needed before it can be called) has its own
timeout [3], but AFAIK that's hard-coded and doesn't have anything to do
with web.timeout.
So, simple question - where's the param used? I hope I've just missed
something, and it's used somewhere....
[1]
https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/LeaderRetrievalHandler.java#L57
[2]
https://github.com/apache/flink/blob/4a78097d0385749daceafd8326930c8cc5f26f1a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L537
[3]
https://github.com/apache/flink/blob/026142d7770afb285d8e73d0aea5939881d3d713/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java#L140
Regards,
Juha
--
Regards,
Juha
Re: web.timeout usage in the code
Posted by Chesnay Schepler <ch...@apache.org>.
Presumably you switched between branches / points in Flinks development,
and either
a) at the commit you are at the flink-rpc module didn't exist yet (but
IntelliJ still displays it because the directory is still present) or
b) IntelliJ hasn't caught up with new modules in which case reloading
the project via maven usually does the trick (open the Maven tool window
-> Reload All Maven Projects).
On 24/08/2021 13:35, Juha Mynttinen wrote:
> I see, that's how it works. Thanks!
>
> Slightly related, which made it a little hard to navigate the code.
> I'm using IntelliJ and for some reason navigating the code in
> flink-rpc doesn't work. It's not listed as a module. In the
> Project-view it is there, but with a different icon than e.g
> flink-runtime. Is there some magic here too?
>
> Juha
>
> On Tue, Aug 24, 2021 at 1:21 PM Chesnay Schepler <chesnay@apache.org
> <ma...@apache.org>> wrote:
>
> There is some internal magic going on. Have a look at the
> interface the handlers work against, you will see that the timeout
> parameter is annotated with @RpcTimeout. The AkkaInvocationHandler
> extracts the timeout based on this annotation and uses it internally.
>
> On 24/08/2021 12:05, Juha Mynttinen wrote:
>> Hey,
>>
>> I was looking at the web.timeout configuration option described
>> as "Timeout for asynchronous operations by the web monitor in
>> milliseconds". I'm interested in where and how it's used internally.
>>
>> I've understood it's the timeout when the Web UI calls something
>> using Rpc (Akka). Correct? Unfortunately, I can't spot where the
>> parameter is actually used so that it would have any effect.
>> Maybe I've missed something?
>>
>> The param is defined in WebOptions and used
>> in RestHandlerConfiguration. The param is stored
>> in LeaderRetrievalHandler [1]. That class merely acts as a holder
>> for the param and e.g. the REST handlers use it. If we look at
>> one of the handlers, e.g. ClusterOverviewHandler, in the end it
>> calls requestClusterOverview in Dispatcher [2]. That has two
>> usages of the timeOut, but if you follow them all the way, the
>> param is unused. I've checked multiple handlers and I couldn't
>> spot any that would really use the param for something.
>>
>> Retrieving the leader (that is needed before it can be called)
>> has its own timeout [3], but AFAIK that's hard-coded and doesn't
>> have anything to do with web.timeout.
>>
>> So, simple question - where's the param used? I hope I've just
>> missed something, and it's used somewhere....
>>
>> [1]
>> https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/LeaderRetrievalHandler.java#L57
>> <https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/LeaderRetrievalHandler.java#L57>
>> [2]
>> https://github.com/apache/flink/blob/4a78097d0385749daceafd8326930c8cc5f26f1a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L537
>> <https://github.com/apache/flink/blob/4a78097d0385749daceafd8326930c8cc5f26f1a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L537>
>> [3]
>> https://github.com/apache/flink/blob/026142d7770afb285d8e73d0aea5939881d3d713/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java#L140
>> <https://github.com/apache/flink/blob/026142d7770afb285d8e73d0aea5939881d3d713/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java#L140>
>>
>> Regards,
>> Juha
>>
>> --
>> Regards,
>> Juha
>
>
>
>
> --
> Regards,
> Juha
Re: web.timeout usage in the code
Posted by Juha Mynttinen <ju...@aiven.io>.
I see, that's how it works. Thanks!
Slightly related, which made it a little hard to navigate the code. I'm
using IntelliJ and for some reason navigating the code in flink-rpc doesn't
work. It's not listed as a module. In the Project-view it is there, but
with a different icon than e.g flink-runtime. Is there some magic here too?
Juha
On Tue, Aug 24, 2021 at 1:21 PM Chesnay Schepler <ch...@apache.org> wrote:
> There is some internal magic going on. Have a look at the interface the
> handlers work against, you will see that the timeout parameter is annotated
> with @RpcTimeout. The AkkaInvocationHandler extracts the timeout based on
> this annotation and uses it internally.
>
> On 24/08/2021 12:05, Juha Mynttinen wrote:
>
> Hey,
>
> I was looking at the web.timeout configuration option described as
> "Timeout for asynchronous operations by the web monitor in milliseconds".
> I'm interested in where and how it's used internally.
>
> I've understood it's the timeout when the Web UI calls something using Rpc
> (Akka). Correct? Unfortunately, I can't spot where the parameter is
> actually used so that it would have any effect. Maybe I've missed something?
>
> The param is defined in WebOptions and used in RestHandlerConfiguration.
> The param is stored in LeaderRetrievalHandler [1]. That class merely acts
> as a holder for the param and e.g. the REST handlers use it. If we look at
> one of the handlers, e.g. ClusterOverviewHandler, in the end it
> calls requestClusterOverview in Dispatcher [2]. That has two usages of the
> timeOut, but if you follow them all the way, the param is unused. I've
> checked multiple handlers and I couldn't spot any that would really use the
> param for something.
>
> Retrieving the leader (that is needed before it can be called) has its own
> timeout [3], but AFAIK that's hard-coded and doesn't have anything to do
> with web.timeout.
>
> So, simple question - where's the param used? I hope I've just missed
> something, and it's used somewhere....
>
> [1]
> https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/LeaderRetrievalHandler.java#L57
> [2]
> https://github.com/apache/flink/blob/4a78097d0385749daceafd8326930c8cc5f26f1a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L537
> [3]
> https://github.com/apache/flink/blob/026142d7770afb285d8e73d0aea5939881d3d713/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java#L140
>
> Regards,
> Juha
>
> --
> Regards,
> Juha
>
>
>
--
Regards,
Juha
Re: web.timeout usage in the code
Posted by Chesnay Schepler <ch...@apache.org>.
There is some internal magic going on. Have a look at the interface the
handlers work against, you will see that the timeout parameter is
annotated with @RpcTimeout. The AkkaInvocationHandler extracts the
timeout based on this annotation and uses it internally.
On 24/08/2021 12:05, Juha Mynttinen wrote:
> Hey,
>
> I was looking at the web.timeout configuration option described as
> "Timeout for asynchronous operations by the web monitor in
> milliseconds". I'm interested in where and how it's used internally.
>
> I've understood it's the timeout when the Web UI calls something using
> Rpc (Akka). Correct? Unfortunately, I can't spot where the parameter
> is actually used so that it would have any effect. Maybe I've missed
> something?
>
> The param is defined in WebOptions and used
> in RestHandlerConfiguration. The param is stored
> in LeaderRetrievalHandler [1]. That class merely acts as a holder for
> the param and e.g. the REST handlers use it. If we look at one of the
> handlers, e.g. ClusterOverviewHandler, in the end it
> calls requestClusterOverview in Dispatcher [2]. That has two usages of
> the timeOut, but if you follow them all the way, the param is unused.
> I've checked multiple handlers and I couldn't spot any that would
> really use the param for something.
>
> Retrieving the leader (that is needed before it can be called) has its
> own timeout [3], but AFAIK that's hard-coded and doesn't have anything
> to do with web.timeout.
>
> So, simple question - where's the param used? I hope I've just missed
> something, and it's used somewhere....
>
> [1]
> https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/LeaderRetrievalHandler.java#L57
> <https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/LeaderRetrievalHandler.java#L57>
> [2]
> https://github.com/apache/flink/blob/4a78097d0385749daceafd8326930c8cc5f26f1a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L537
> <https://github.com/apache/flink/blob/4a78097d0385749daceafd8326930c8cc5f26f1a/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L537>
> [3]
> https://github.com/apache/flink/blob/026142d7770afb285d8e73d0aea5939881d3d713/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java#L140
> <https://github.com/apache/flink/blob/026142d7770afb285d8e73d0aea5939881d3d713/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java#L140>
>
> Regards,
> Juha
>
> --
> Regards,
> Juha