You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "clarax (via GitHub)" <gi...@apache.org> on 2024/08/22 23:12:24 UTC

[PR] [FLINK-36140] Log a warning when pods are terminated by kubernetes [flink]

clarax opened a new pull request, #25242:
URL: https://github.com/apache/flink/pull/25242

   ## What is the purpose of the change
   
   Scheduled maintenance or buggy nodes on Kubernetes can result random pod termination and eventually a series of job restarts due to rolling restart of the Kubernetes cluster nodes. The larger the job is the higher the chance it is affected. The jobs should be able to auto-recover from these issues, but can cause unwanted turbulence in large scale pipeline.
   In this case, it is very difficult to identify what is causing the restarts without knowing the issue at Kubernetes layer and the keyword to search with because it is logged at INFO level.
   We need to log this at higher level. If changing it from INFO to ERROR breaks monitoring we should at least log as warning. 
   
   ## Brief change log
   
     - Change the log level from INFO to WARN
   
   
   ## Verifying this change
   
   
   This change added tests and can be verified as follows:
   
     - *Manually verified the change at local 
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? No
     - If yes, how is the feature documented? not applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-36140] Log a warning when pods are terminated by kubernetes [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #25242:
URL: https://github.com/apache/flink/pull/25242#issuecomment-2305900755

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1c186c266462faeeb0bd3f1cb89a6bd6b38ab98d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1c186c266462faeeb0bd3f1cb89a6bd6b38ab98d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c186c266462faeeb0bd3f1cb89a6bd6b38ab98d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-36140] Log a warning when pods are terminated by kubernetes [flink]

Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi merged PR #25242:
URL: https://github.com/apache/flink/pull/25242


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-36140] Log a warning when pods are terminated by kubernetes [flink]

Posted by "gaborgsomogyi (via GitHub)" <gi...@apache.org>.
gaborgsomogyi commented on PR #25242:
URL: https://github.com/apache/flink/pull/25242#issuecomment-2306470908

   Unless there are further comments I'm intended to merge this on Monday.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org