You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Sharon Xie <sh...@gmail.com> on 2021/12/29 19:24:09 UTC

Re: Unable to update logback configuration in Flink Native Kubernetes

I've faced the same issue before.

I figured out that there is an internal configuration
`$internal.deployment.config-dir` (code
<https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
which allows me to specify a local folder which contains the logback config
using file `logback-console.xml`. The content of the file is then used to
create the config map.

Hope it helps.


Sharon

On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
wrote:

> Hi
>
> I have created a Flink Native Kubernetes (1.14.2) cluster which is
> successful. I am trying to update the logback configuration for which I am
> using the configmap exposed by Flink Native Kubernetes. Flink Native
> Kubernetes is creating this configmap during the start of the cluster and
> deleting it when the cluster is stopped and this behavior is as per the
> official documentation.
>
> I updated the logback configmap which is also successful and this process
> even updates the actual logback files (conf folder) in the job manager and
> task manager. But Flink is not loading (hot reloading) this logback
> configuration.
>
> Also I want to make sure that the logback configmap configuration is
> persisted even during cluster restarts. But the Flink Native Kubernetes
> recreates the configmap each time the cluster is started.
>
> What is that I am missing here? How to make the updated logback
> configuration work?
>
>
> Thanks & Regards
> Raghavendar T S
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

Re: Unable to update logback configuration in Flink Native Kubernetes

Posted by Yang Wang <da...@gmail.com>.
Sorry for the late reply.

Flink clients will ship the log4j-console.properties and
logback-console.xml via K8s ConfigMap and then mount to
JobManager/TaskManager pod.
So if you want to update the log settings or using logback, all you need is
to update the client-local files.

Best,
Yang




Raghavendar T S <ra...@gmail.com> 于2021年12月31日周五 12:47写道：

> Hi Sharon
>
> Thanks a lot. I just updated the files (flink-conf.yaml and
> logback-console.xml) in the local conf folder and it worked as expected.
>
> Thanks & Regards
> Raghavendar T S
> MERAS Plugins
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_-8574539363514296023_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Thu, Dec 30, 2021 at 12:54 AM Sharon Xie <sh...@gmail.com>
> wrote:
>
>> I've faced the same issue before.
>>
>> I figured out that there is an internal configuration
>> `$internal.deployment.config-dir` (code
>> <https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
>> which allows me to specify a local folder which contains the logback config
>> using file `logback-console.xml`. The content of the file is then used to
>> create the config map.
>>
>> Hope it helps.
>>
>>
>> Sharon
>>
>> On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I have created a Flink Native Kubernetes (1.14.2) cluster which is
>>> successful. I am trying to update the logback configuration for which I am
>>> using the configmap exposed by Flink Native Kubernetes. Flink Native
>>> Kubernetes is creating this configmap during the start of the cluster and
>>> deleting it when the cluster is stopped and this behavior is as per the
>>> official documentation.
>>>
>>> I updated the logback configmap which is also successful and this
>>> process even updates the actual logback files (conf folder) in the job
>>> manager and task manager. But Flink is not loading (hot reloading) this
>>> logback configuration.
>>>
>>> Also I want to make sure that the logback configmap configuration is
>>> persisted even during cluster restarts. But the Flink Native Kubernetes
>>> recreates the configmap each time the cluster is started.
>>>
>>> What is that I am missing here? How to make the updated logback
>>> configuration work?
>>>
>>>
>>> Thanks & Regards
>>> Raghavendar T S
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
>>> www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> <#m_-8574539363514296023_m_-6539971309794987579_m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>
>
> --
> Raghavendar T S
> www.teknosrc.com
>

Re: Unable to update logback configuration in Flink Native Kubernetes

Posted by Yang Wang <da...@gmail.com>.
Sorry for the late reply.

Flink clients will ship the log4j-console.properties and
logback-console.xml via K8s ConfigMap and then mount to
JobManager/TaskManager pod.
So if you want to update the log settings or using logback, all you need is
to update the client-local files.

Best,
Yang




Raghavendar T S <ra...@gmail.com> 于2021年12月31日周五 12:47写道：

> Hi Sharon
>
> Thanks a lot. I just updated the files (flink-conf.yaml and
> logback-console.xml) in the local conf folder and it worked as expected.
>
> Thanks & Regards
> Raghavendar T S
> MERAS Plugins
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#m_-8574539363514296023_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Thu, Dec 30, 2021 at 12:54 AM Sharon Xie <sh...@gmail.com>
> wrote:
>
>> I've faced the same issue before.
>>
>> I figured out that there is an internal configuration
>> `$internal.deployment.config-dir` (code
>> <https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
>> which allows me to specify a local folder which contains the logback config
>> using file `logback-console.xml`. The content of the file is then used to
>> create the config map.
>>
>> Hope it helps.
>>
>>
>> Sharon
>>
>> On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I have created a Flink Native Kubernetes (1.14.2) cluster which is
>>> successful. I am trying to update the logback configuration for which I am
>>> using the configmap exposed by Flink Native Kubernetes. Flink Native
>>> Kubernetes is creating this configmap during the start of the cluster and
>>> deleting it when the cluster is stopped and this behavior is as per the
>>> official documentation.
>>>
>>> I updated the logback configmap which is also successful and this
>>> process even updates the actual logback files (conf folder) in the job
>>> manager and task manager. But Flink is not loading (hot reloading) this
>>> logback configuration.
>>>
>>> Also I want to make sure that the logback configmap configuration is
>>> persisted even during cluster restarts. But the Flink Native Kubernetes
>>> recreates the configmap each time the cluster is started.
>>>
>>> What is that I am missing here? How to make the updated logback
>>> configuration work?
>>>
>>>
>>> Thanks & Regards
>>> Raghavendar T S
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
>>> www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>>> <#m_-8574539363514296023_m_-6539971309794987579_m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>
>
> --
> Raghavendar T S
> www.teknosrc.com
>

Re: Unable to update logback configuration in Flink Native Kubernetes

Posted by Raghavendar T S <ra...@gmail.com>.
Hi Sharon

Thanks a lot. I just updated the files (flink-conf.yaml and
logback-console.xml) in the local conf folder and it worked as expected.

Thanks & Regards
Raghavendar T S
MERAS Plugins

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Thu, Dec 30, 2021 at 12:54 AM Sharon Xie <sh...@gmail.com> wrote:

> I've faced the same issue before.
>
> I figured out that there is an internal configuration
> `$internal.deployment.config-dir` (code
> <https://github.com/apache/flink/blob/3071f5c8471b31244642291cee769aeb55733079/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptionsInternal.java>)
> which allows me to specify a local folder which contains the logback config
> using file `logback-console.xml`. The content of the file is then used to
> create the config map.
>
> Hope it helps.
>
>
> Sharon
>
> On Wed, Dec 29, 2021 at 7:04 AM Raghavendar T S <ra...@gmail.com>
> wrote:
>
>> Hi
>>
>> I have created a Flink Native Kubernetes (1.14.2) cluster which is
>> successful. I am trying to update the logback configuration for which I am
>> using the configmap exposed by Flink Native Kubernetes. Flink Native
>> Kubernetes is creating this configmap during the start of the cluster and
>> deleting it when the cluster is stopped and this behavior is as per the
>> official documentation.
>>
>> I updated the logback configmap which is also successful and this process
>> even updates the actual logback files (conf folder) in the job manager and
>> task manager. But Flink is not loading (hot reloading) this logback
>> configuration.
>>
>> Also I want to make sure that the logback configmap configuration is
>> persisted even during cluster restarts. But the Flink Native Kubernetes
>> recreates the configmap each time the cluster is started.
>>
>> What is that I am missing here? How to make the updated logback
>> configuration work?
>>
>>
>> Thanks & Regards
>> Raghavendar T S
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
>> www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
>> <#m_-6539971309794987579_m_9211879584941238630_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>

-- 
Raghavendar T S
www.teknosrc.com