You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2019/04/24 16:02:16 UTC
QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )
Hello folks,
Following
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state
.
for setting up the Queryable Server and proxy, I have my classpath ( the
lib directory ) that has the required jar, But I do not see the mentioned
log and of course am not able to set up the QS server/Proxy . This has
worked on 1.7.2 and I think I have everything as advised, see the logs
below. I do not see this log "Started the Queryable State Proxy Server @
...". Any one with this issue...
2019-04-24 15:54:26,296 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- -Dtaskmanager.numberOfTaskSlots=1
2019-04-24 15:54:26,296 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- --configDir
2019-04-24 15:54:26,296 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- /usr/local/flink/conf
2019-04-24 15:54:26,296 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- Classpath: /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:
*/usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*
:/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
2019-04-24 15:54:26,296 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
-
--------------------------------------------------------------------------------
2019-04-24 15:54:26,298 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- Registered UNIX signal handlers for [TERM, HUP, INT]
2019-04-24 15:54:26,300 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner
- Maximum number of open file descriptors is 65536.
2019-04-24 15:54:26,305 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.backend.fs.checkpointdir,
hdfs:///flink-checkpoints_k8s_test/prod
2
Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )
Posted by Ufuk Celebi <uc...@apache.org>.
Actually, I couldn't even find a mention of this flag in the docs here:
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/queryable_state.html
– Ufuk
On Mon, Apr 29, 2019 at 8:45 AM Ufuk Celebi <uc...@apache.org> wrote:
> I didn't find this as part of the
> https://flink.apache.org/news/2019/04/09/release-1.8.0.html notes.
>
> I think an update to the Important Changes section would be valuable for
> users upgrading to 1.8 from earlier releases. Also, logging that the
> library is on the classpath but the feature flag is set to false would be a
> helpful.
>
> – Ufuk
>
>
> On Thu, Apr 25, 2019 at 4:13 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> Ditto that, queryable-state.enable to true works.
>>
>> Thanks everyone.
>>
>> On Thu, Apr 25, 2019 at 6:28 AM Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>>> Hi Vishal,
>>>
>>> As Guowei mentioned you have to enable the Queryable state. The default
>>> setting was changed in 1.8.0. There is an open JIRA[1] for changing the
>>> documentation accordingly.
>>>
>>> Best,
>>>
>>> Dawid
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-12274
>>> On 25/04/2019 03:27, Guowei Ma wrote:
>>>
>>> You could try to set queryable-state.enable to true. And check again.
>>>
>>> Vishal Santoshi <vi...@gmail.com>于2019年4月25日 周四上午1:40写道：
>>>
>>>> Any one ?
>>>>
>>>> On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi <
>>>> vishal.santoshi@gmail.com> wrote:
>>>>
>>>>> Hello folks,
>>>>>
>>>>> Following
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state .
>>>>> for setting up the Queryable Server and proxy, I have my classpath ( the
>>>>> lib directory ) that has the required jar, But I do not see the mentioned
>>>>> log and of course am not able to set up the QS server/Proxy . This has
>>>>> worked on 1.7.2 and I think I have everything as advised, see the logs
>>>>> below. I do not see this log "Started the Queryable State Proxy
>>>>> Server @ ...". Any one with this issue...
>>>>>
>>>>>
>>>>>
>>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> - -Dtaskmanager.numberOfTaskSlots=1
>>>>>
>>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> - --configDir
>>>>>
>>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> - /usr/local/flink/conf
>>>>>
>>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> - Classpath:
>>>>> /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:
>>>>> */usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*
>>>>> :/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
>>>>>
>>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> -
>>>>> --------------------------------------------------------------------------------
>>>>>
>>>>> 2019-04-24 15:54:26,298 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> - Registered UNIX signal handlers for [TERM, HUP, INT]
>>>>>
>>>>> 2019-04-24 15:54:26,300 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>>> - Maximum number of open file descriptors is 65536.
>>>>>
>>>>> 2019-04-24 15:54:26,305 INFO
>>>>> org.apache.flink.configuration.GlobalConfiguration -
>>>>> Loading configuration property: state.backend.fs.checkpointdir,
>>>>> hdfs:///flink-checkpoints_k8s_test/prod
>>>>>
>>>>> 2
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>> Best,
>>> Guowei
>>>
>>>
Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )
Posted by Ufuk Celebi <uc...@apache.org>.
I didn't find this as part of the
https://flink.apache.org/news/2019/04/09/release-1.8.0.html notes.
I think an update to the Important Changes section would be valuable for
users upgrading to 1.8 from earlier releases. Also, logging that the
library is on the classpath but the feature flag is set to false would be a
helpful.
– Ufuk
On Thu, Apr 25, 2019 at 4:13 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Ditto that, queryable-state.enable to true works.
>
> Thanks everyone.
>
> On Thu, Apr 25, 2019 at 6:28 AM Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
>> Hi Vishal,
>>
>> As Guowei mentioned you have to enable the Queryable state. The default
>> setting was changed in 1.8.0. There is an open JIRA[1] for changing the
>> documentation accordingly.
>>
>> Best,
>>
>> Dawid
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-12274
>> On 25/04/2019 03:27, Guowei Ma wrote:
>>
>> You could try to set queryable-state.enable to true. And check again.
>>
>> Vishal Santoshi <vi...@gmail.com>于2019年4月25日 周四上午1:40写道：
>>
>>> Any one ?
>>>
>>> On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> Hello folks,
>>>>
>>>> Following
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state .
>>>> for setting up the Queryable Server and proxy, I have my classpath ( the
>>>> lib directory ) that has the required jar, But I do not see the mentioned
>>>> log and of course am not able to set up the QS server/Proxy . This has
>>>> worked on 1.7.2 and I think I have everything as advised, see the logs
>>>> below. I do not see this log "Started the Queryable State Proxy
>>>> Server @ ...". Any one with this issue...
>>>>
>>>>
>>>>
>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> - -Dtaskmanager.numberOfTaskSlots=1
>>>>
>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> - --configDir
>>>>
>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> - /usr/local/flink/conf
>>>>
>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> - Classpath:
>>>> /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:
>>>> */usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*
>>>> :/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
>>>>
>>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> -
>>>> --------------------------------------------------------------------------------
>>>>
>>>> 2019-04-24 15:54:26,298 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> - Registered UNIX signal handlers for [TERM, HUP, INT]
>>>>
>>>> 2019-04-24 15:54:26,300 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>>> - Maximum number of open file descriptors is 65536.
>>>>
>>>> 2019-04-24 15:54:26,305 INFO
>>>> org.apache.flink.configuration.GlobalConfiguration -
>>>> Loading configuration property: state.backend.fs.checkpointdir,
>>>> hdfs:///flink-checkpoints_k8s_test/prod
>>>>
>>>> 2
>>>>
>>>>
>>>>
>>>>
>>>> --
>> Best,
>> Guowei
>>
>>
Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )
Posted by Vishal Santoshi <vi...@gmail.com>.
Ditto that, queryable-state.enable to true works.
Thanks everyone.
On Thu, Apr 25, 2019 at 6:28 AM Dawid Wysakowicz <dw...@apache.org>
wrote:
> Hi Vishal,
>
> As Guowei mentioned you have to enable the Queryable state. The default
> setting was changed in 1.8.0. There is an open JIRA[1] for changing the
> documentation accordingly.
>
> Best,
>
> Dawid
>
> [1] https://issues.apache.org/jira/browse/FLINK-12274
> On 25/04/2019 03:27, Guowei Ma wrote:
>
> You could try to set queryable-state.enable to true. And check again.
>
> Vishal Santoshi <vi...@gmail.com>于2019年4月25日 周四上午1:40写道：
>
>> Any one ?
>>
>> On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>>> Hello folks,
>>>
>>> Following
>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state .
>>> for setting up the Queryable Server and proxy, I have my classpath ( the
>>> lib directory ) that has the required jar, But I do not see the mentioned
>>> log and of course am not able to set up the QS server/Proxy . This has
>>> worked on 1.7.2 and I think I have everything as advised, see the logs
>>> below. I do not see this log "Started the Queryable State Proxy
>>> Server @ ...". Any one with this issue...
>>>
>>>
>>>
>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> - -Dtaskmanager.numberOfTaskSlots=1
>>>
>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> - --configDir
>>>
>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> - /usr/local/flink/conf
>>>
>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> - Classpath:
>>> /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:
>>> */usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*
>>> :/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
>>>
>>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> -
>>> --------------------------------------------------------------------------------
>>>
>>> 2019-04-24 15:54:26,298 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> - Registered UNIX signal handlers for [TERM, HUP, INT]
>>>
>>> 2019-04-24 15:54:26,300 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>>> - Maximum number of open file descriptors is 65536.
>>>
>>> 2019-04-24 15:54:26,305 INFO
>>> org.apache.flink.configuration.GlobalConfiguration - Loading
>>> configuration property: state.backend.fs.checkpointdir,
>>> hdfs:///flink-checkpoints_k8s_test/prod
>>>
>>> 2
>>>
>>>
>>>
>>>
>>> --
> Best,
> Guowei
>
>
Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2
)
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi Vishal,
As Guowei mentioned you have to enable the Queryable state. The default
setting was changed in 1.8.0. There is an open JIRA[1] for changing the
documentation accordingly.
Best,
Dawid
[1] https://issues.apache.org/jira/browse/FLINK-12274
On 25/04/2019 03:27, Guowei Ma wrote:
> You could try to set queryable-state.enable to true. And check again.
>
> Vishal Santoshi <vishal.santoshi@gmail.com
> <ma...@gmail.com>>于2019年4月25日 周四上午1:40写道：
>
> Any one ?
>
> On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi
> <vishal.santoshi@gmail.com <ma...@gmail.com>> wrote:
>
> Hello folks,
>
>
> Following https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state .
> for setting up the Queryable Server and proxy, I have my
> classpath ( the lib directory ) that has the required jar,
> But I do not see the mentioned log and of course am not able
> to set up the QS server/Proxy . This has worked on 1.7.2 and I
> think I have everything as advised, see the logs below. I do
> not see this log |"Started the Queryable State Proxy Server
> @ ..."|. Any one with this issue...
>
>
>
> 2019-04-24 15:54:26,296 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - -Dtaskmanager.numberOfTaskSlots=1
>
> 2019-04-24 15:54:26,296 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - --configDir
>
> 2019-04-24 15:54:26,296 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - /usr/local/flink/conf
>
> 2019-04-24 15:54:26,296 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - Classpath:
> /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:*/usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*:/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
>
> 2019-04-24 15:54:26,296 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> -
> --------------------------------------------------------------------------------
>
> 2019-04-24 15:54:26,298 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - Registered UNIX signal handlers for [TERM, HUP, INT]
>
> 2019-04-24 15:54:26,300 INFO
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - Maximum number of open file descriptors is 65536.
>
> 2019-04-24 15:54:26,305 INFO
> org.apache.flink.configuration.GlobalConfiguration
> - Loading configuration property:
> state.backend.fs.checkpointdir,
> hdfs:///flink-checkpoints_k8s_test/prod
>
> 2
>
>
>
>
>
> --
> Best,
> Guowei
Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )
Posted by Guowei Ma <gu...@gmail.com>.
You could try to set queryable-state.enable to true. And check again.
Vishal Santoshi <vi...@gmail.com>于2019年4月25日 周四上午1:40写道：
> Any one ?
>
> On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi <
> vishal.santoshi@gmail.com> wrote:
>
>> Hello folks,
>>
>> Following
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state .
>> for setting up the Queryable Server and proxy, I have my classpath ( the
>> lib directory ) that has the required jar, But I do not see the mentioned
>> log and of course am not able to set up the QS server/Proxy . This has
>> worked on 1.7.2 and I think I have everything as advised, see the logs
>> below. I do not see this log "Started the Queryable State Proxy Server
>> @ ...". Any one with this issue...
>>
>>
>>
>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> - -Dtaskmanager.numberOfTaskSlots=1
>>
>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> - --configDir
>>
>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> - /usr/local/flink/conf
>>
>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> - Classpath:
>> /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:
>> */usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*
>> :/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
>>
>> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> -
>> --------------------------------------------------------------------------------
>>
>> 2019-04-24 15:54:26,298 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> - Registered UNIX signal handlers for [TERM, HUP, INT]
>>
>> 2019-04-24 15:54:26,300 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
>> - Maximum number of open file descriptors is 65536.
>>
>> 2019-04-24 15:54:26,305 INFO
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: state.backend.fs.checkpointdir,
>> hdfs:///flink-checkpoints_k8s_test/prod
>>
>> 2
>>
>>
>>
>>
>> --
Best,
Guowei
Re: QueryableState startup regression in 1.8.0 ( migration from 1.7.2 )
Posted by Vishal Santoshi <vi...@gmail.com>.
Any one ?
On Wed, Apr 24, 2019 at 12:02 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Hello folks,
>
> Following
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#querying-state .
> for setting up the Queryable Server and proxy, I have my classpath ( the
> lib directory ) that has the required jar, But I do not see the mentioned
> log and of course am not able to set up the QS server/Proxy . This has
> worked on 1.7.2 and I think I have everything as advised, see the logs
> below. I do not see this log "Started the Queryable State Proxy Server
> @ ...". Any one with this issue...
>
>
>
> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - -Dtaskmanager.numberOfTaskSlots=1
>
> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - --configDir
>
> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - /usr/local/flink/conf
>
> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - Classpath:
> /usr/local/flink/lib/flink-metrics-prometheus-1.8.0.jar:
> */usr/local/flink/lib/flink-queryable-state-runtime_2.11-1.8.0.jar*
> :/usr/local/flink/lib/hadoop.jar:/usr/local/flink/lib/jobs.jar:/usr/local/flink/lib/log4j-1.2.17.jar:/usr/local/flink/lib/slf4j-log4j12-1.7.15.jar:/usr/local/flink/lib/flink-dist_2.11-1.8.0.jar:::
>
> 2019-04-24 15:54:26,296 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> -
> --------------------------------------------------------------------------------
>
> 2019-04-24 15:54:26,298 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - Registered UNIX signal handlers for [TERM, HUP, INT]
>
> 2019-04-24 15:54:26,300 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner
> - Maximum number of open file descriptors is 65536.
>
> 2019-04-24 15:54:26,305 INFO
> org.apache.flink.configuration.GlobalConfiguration - Loading
> configuration property: state.backend.fs.checkpointdir,
> hdfs:///flink-checkpoints_k8s_test/prod
>
> 2
>
>
>
>
>