You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yiheng tang (Jira)" <ji...@apache.org> on 2024/07/29 07:39:00 UTC
[jira] [Comment Edited] (FLINK-35898) When running Flink on Yarn, if the log file path contains a space, the startup will fail.
[ https://issues.apache.org/jira/browse/FLINK-35898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869254#comment-17869254 ]
yiheng tang edited comment on FLINK-35898 at 7/29/24 7:38 AM:
--------------------------------------------------------------
If my log.file named as "/data08/yarn/userlogs/application 1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log", the command will be incorrectly truncated.
was (Author: JIRAUSER306337):
If my log.file named as "/data08/yarn/userlogs/application 1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log", the command is incorrectly truncated.
> When running Flink on Yarn, if the log file path contains a space, the startup will fail.
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-35898
> URL: https://issues.apache.org/jira/browse/FLINK-35898
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.20.0
> Reporter: yiheng tang
> Priority: Minor
> Attachments: image-2024-07-25-21-51-50-072.png, image-2024-07-25-21-52-26-973.png
>
>
> We use Yarn to launch Flink jobs and try to add custom parameters to configure the log output format in the YarnLogConfigUtil.getLog4jCommand method. However, when the parameter value contains spaces, it causes the startup to fail.
> The start command in the Yarn container is as follows, omitting some irrelevant parameters.
> {code:java}
> /bin/bash -c "$JAVA_HOME/bin/java -Dlog.file="/data08/yarn/userlogs/application_1719804259110_16060/container_e43_1719804259110_16060_01_000001/jobmanager.log" -Dlog4j.configuration=file.properties -Dlog4j.configurationFile=file.properties -Dlog4j2.isThreadContextMapInheritable=true -Dlog.layout.pattern="%d{yyyy-MM-dd HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO" {code}
> We discovered that when the value of log.layout.pattern contains spaces, the command is incorrectly truncated, causing the content after the space to be unrecognized, and therefore the Flink job cannot be launched in the Yarn container.
> {code:java}
> {HH:mm} %-5p %-60c %-60t %x - %m%n %ex" -Dlog.level=INFO"{code}
> Therefore, we suggest using single quotes (') instead of double quotes (") to wrap the values of parameters in the {{getLogBackCommand}} method and {{getLog4jCommand}} method of the {{YarnLogConfigUtil}} class.
> {code:java}
> private static String getLogBackCommand(final String logConfigFilePath) {
> final boolean hasLogback = logConfigFilePath.endsWith(CONFIG_FILE_LOGBACK_NAME);
> if (!hasLogback) {
> return "";
> } return new StringBuilder(
> "-Dlog.file=\""
> + ApplicationConstants.LOG_DIR_EXPANSION_VAR
> + "/jobmanager.log\"")
> .append(" -Dlogback.configurationFile=file:" + CONFIG_FILE_LOGBACK_NAME)
> .toString();
> } private static String getLog4jCommand(final String logConfigFilePath) {
> final boolean hasLog4j = logConfigFilePath.endsWith(CONFIG_FILE_LOG4J_NAME);
> if (!hasLog4j) {
> return "";
> } return new StringBuilder(
> "-Dlog.file=\""
> + ApplicationConstants.LOG_DIR_EXPANSION_VAR
> + "/jobmanager.log\"")
> .append(" -Dlog4j.configuration=file:" + CONFIG_FILE_LOG4J_NAME)
> .append(" -Dlog4j.configurationFile=file:" + CONFIG_FILE_LOG4J_NAME)
> .toString();
> }
> } {code}
> ---
> !image-2024-07-25-21-52-26-973.png|width=843,height=506!!image-2024-07-25-21-51-50-072.png|width=417,height=218!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)