You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Raghavendar T S <ra...@gmail.com> on 2021/12/29 15:04:25 UTC
Unable to update logback configuration in Flink Native Kubernetes
Hi
I have created a Flink Native Kubernetes (1.14.2) cluster which is
successful. I am trying to update the logback configuration for which I am
using the configmap exposed by Flink Native Kubernetes. Flink Native
Kubernetes is creating this configmap during the start of the cluster and
deleting it when the cluster is stopped and this behavior is as per the
official documentation.
I updated the logback configmap which is also successful and this process
even updates the actual logback files (conf folder) in the job manager and
task manager. But Flink is not loading (hot reloading) this logback
configuration.
Also I want to make sure that the logback configmap configuration is
persisted even during cluster restarts. But the Flink Native Kubernetes
recreates the configmap each time the cluster is started.
What is that I am missing here? How to make the updated logback
configuration work?
Thanks & Regards
Raghavendar T S
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: Unable to update logback configuration in Flink Native Kubernetes
Posted by Yang Wang <da...@gmail.com>.
Sorry for the late reply.
Flink clients will ship the log4j-console.properties and
logback-console.xml via K8s ConfigMap and then mount to
JobManager/TaskManager pod.
So if you want to update the log settings or using logback, all you need is
to update the client-local files.
Best,
Yang
Raghavendar T S <ra...@gmail.com> 于2021年12月31日周五 12:47写道：
> Hi Sharon
>
> Thanks a lot. I just updated the files (flink-conf.yaml and
> logback-console.xml) in the local conf folder and it worked as expected.
>
> Thanks & Regards
> Raghavendar T S
> MERAS Plugins
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_-8574539363514296023_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Thu, Dec 30, 2021 at 12:54 AM Sharon Xie <sh...@gmail.com>
> wrote:
>
>> I've faced the same issue before.
>>
>> I figured out that there is an internal configuration
>> `$internal.deployment.config-dir` (code
>> <https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
>> which allows me to specify a local folder which contains the logback config
>> using file `logback-console.xml`. The content of the file is then used to
>> create the config map.
>>
>> Hope it helps.
>>
>>
>> Sharon
>>
>> On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I have created a Flink Native Kubernetes (1.14.2) cluster which is
>>> successful. I am trying to update the logback configuration for which I am
>>> using the configmap exposed by Flink Native Kubernetes. Flink Native
>>> Kubernetes is creating this configmap during the start of the cluster and
>>> deleting it when the cluster is stopped and this behavior is as per the
>>> official documentation.
>>>
>>> I updated the logback configmap which is also successful and this
>>> process even updates the actual logback files (conf folder) in the job
>>> manager and task manager. But Flink is not loading (hot reloading) this
>>> logback configuration.
>>>
>>> Also I want to make sure that the logback configmap configuration is
>>> persisted even during cluster restarts. But the Flink Native Kubernetes
>>> recreates the configmap each time the cluster is started.
>>>
>>> What is that I am missing here? How to make the updated logback
>>> configuration work?
>>>
>>>
>>> Thanks & Regards
>>> Raghavendar T S
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
>>> www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> <#m_-8574539363514296023_m_-6539971309794987579_m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>
>
> --
> Raghavendar T S
> www.teknosrc.com
>
Re: Unable to update logback configuration in Flink Native Kubernetes
Posted by Yang Wang <da...@gmail.com>.
Sorry for the late reply.
Flink clients will ship the log4j-console.properties and
logback-console.xml via K8s ConfigMap and then mount to
JobManager/TaskManager pod.
So if you want to update the log settings or using logback, all you need is
to update the client-local files.
Best,
Yang
Raghavendar T S <ra...@gmail.com> 于2021年12月31日周五 12:47写道：
> Hi Sharon
>
> Thanks a lot. I just updated the files (flink-conf.yaml and
> logback-console.xml) in the local conf folder and it worked as expected.
>
> Thanks & Regards
> Raghavendar T S
> MERAS Plugins
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_-8574539363514296023_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Thu, Dec 30, 2021 at 12:54 AM Sharon Xie <sh...@gmail.com>
> wrote:
>
>> I've faced the same issue before.
>>
>> I figured out that there is an internal configuration
>> `$internal.deployment.config-dir` (code
>> <https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
>> which allows me to specify a local folder which contains the logback config
>> using file `logback-console.xml`. The content of the file is then used to
>> create the config map.
>>
>> Hope it helps.
>>
>>
>> Sharon
>>
>> On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I have created a Flink Native Kubernetes (1.14.2) cluster which is
>>> successful. I am trying to update the logback configuration for which I am
>>> using the configmap exposed by Flink Native Kubernetes. Flink Native
>>> Kubernetes is creating this configmap during the start of the cluster and
>>> deleting it when the cluster is stopped and this behavior is as per the
>>> official documentation.
>>>
>>> I updated the logback configmap which is also successful and this
>>> process even updates the actual logback files (conf folder) in the job
>>> manager and task manager. But Flink is not loading (hot reloading) this
>>> logback configuration.
>>>
>>> Also I want to make sure that the logback configmap configuration is
>>> persisted even during cluster restarts. But the Flink Native Kubernetes
>>> recreates the configmap each time the cluster is started.
>>>
>>> What is that I am missing here? How to make the updated logback
>>> configuration work?
>>>
>>>
>>> Thanks & Regards
>>> Raghavendar T S
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
>>> www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> <#m_-8574539363514296023_m_-6539971309794987579_m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>
>
> --
> Raghavendar T S
> www.teknosrc.com
>
Re: Unable to update logback configuration in Flink Native Kubernetes
Posted by Raghavendar T S <ra...@gmail.com>.
Hi Sharon
Thanks a lot. I just updated the files (flink-conf.yaml and
logback-console.xml) in the local conf folder and it worked as expected.
Thanks & Regards
Raghavendar T S
MERAS Plugins
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Thu, Dec 30, 2021 at 12:54 AM Sharon Xie <sh...@gmail.com> wrote:
> I've faced the same issue before.
>
> I figured out that there is an internal configuration
> `$internal.deployment.config-dir` (code
> <https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
> which allows me to specify a local folder which contains the logback config
> using file `logback-console.xml`. The content of the file is then used to
> create the config map.
>
> Hope it helps.
>
>
> Sharon
>
> On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
> wrote:
>
>> Hi
>>
>> I have created a Flink Native Kubernetes (1.14.2) cluster which is
>> successful. I am trying to update the logback configuration for which I am
>> using the configmap exposed by Flink Native Kubernetes. Flink Native
>> Kubernetes is creating this configmap during the start of the cluster and
>> deleting it when the cluster is stopped and this behavior is as per the
>> official documentation.
>>
>> I updated the logback configmap which is also successful and this process
>> even updates the actual logback files (conf folder) in the job manager and
>> task manager. But Flink is not loading (hot reloading) this logback
>> configuration.
>>
>> Also I want to make sure that the logback configmap configuration is
>> persisted even during cluster restarts. But the Flink Native Kubernetes
>> recreates the configmap each time the cluster is started.
>>
>> What is that I am missing here? How to make the updated logback
>> configuration work?
>>
>>
>> Thanks & Regards
>> Raghavendar T S
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
>> www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>> <#m_-6539971309794987579_m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>
--
Raghavendar T S
www.teknosrc.com
Re: Unable to update logback configuration in Flink Native Kubernetes
Posted by Sharon Xie <sh...@gmail.com>.
I've faced the same issue before.
I figured out that there is an internal configuration
`$internal.deployment.config-dir` (code
<https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
which allows me to specify a local folder which contains the logback config
using file `logback-console.xml`. The content of the file is then used to
create the config map.
Hope it helps.
Sharon
On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
wrote:
> Hi
>
> I have created a Flink Native Kubernetes (1.14.2) cluster which is
> successful. I am trying to update the logback configuration for which I am
> using the configmap exposed by Flink Native Kubernetes. Flink Native
> Kubernetes is creating this configmap during the start of the cluster and
> deleting it when the cluster is stopped and this behavior is as per the
> official documentation.
>
> I updated the logback configmap which is also successful and this process
> even updates the actual logback files (conf folder) in the job manager and
> task manager. But Flink is not loading (hot reloading) this logback
> configuration.
>
> Also I want to make sure that the logback configmap configuration is
> persisted even during cluster restarts. But the Flink Native Kubernetes
> recreates the configmap each time the cluster is started.
>
> What is that I am missing here? How to make the updated logback
> configuration work?
>
>
> Thanks & Regards
> Raghavendar T S
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
Re: Unable to update logback configuration in Flink Native Kubernetes
Posted by Sharon Xie <sh...@gmail.com>.
I've faced the same issue before.
I figured out that there is an internal configuration
`$internal.deployment.config-dir` (code
<https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
which allows me to specify a local folder which contains the logback config
using file `logback-console.xml`. The content of the file is then used to
create the config map.
Hope it helps.
Sharon
On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
wrote:
> Hi
>
> I have created a Flink Native Kubernetes (1.14.2) cluster which is
> successful. I am trying to update the logback configuration for which I am
> using the configmap exposed by Flink Native Kubernetes. Flink Native
> Kubernetes is creating this configmap during the start of the cluster and
> deleting it when the cluster is stopped and this behavior is as per the
> official documentation.
>
> I updated the logback configmap which is also successful and this process
> even updates the actual logback files (conf folder) in the job manager and
> task manager. But Flink is not loading (hot reloading) this logback
> configuration.
>
> Also I want to make sure that the logback configmap configuration is
> persisted even during cluster restarts. But the Flink Native Kubernetes
> recreates the configmap each time the cluster is started.
>
> What is that I am missing here? How to make the updated logback
> configuration work?
>
>
> Thanks & Regards
> Raghavendar T S
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>