[jira] [Commented] (FLINK-15519) Preserve logs from BashJavaUtils and make them part of TM logs

["Andrey Zagrebin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010884#comment-17010884 ] 

Andrey Zagrebin commented on FLINK-15519:
-----------------------------------------

The latest suggestion from a joint offline discussion:
 * build BashJavaUtils as a separate jar, packed with the log4j binder and its fixed configuration
 * the fixed logging configuration should point the logs to the stdout and output minimal info level log strings, possibly w/o any prefixes
 * put into e.g. /bin directory in flink-dist (not into lib to exclude BashJavaUtils and its logging setup from  the Flink class path)
 * form the class path for running BashJavaUtils from its jar and lib/flink-dist*.jar which should not have any binders
 * save stdout of BashJavaUtils run into an env var of the parent bash process
 * the output has to be tailed to some reasonable number of lines like 1000 to avoid env var max size overflow
 * tail the last line for the result
 * read the log lines from env var in the subsequently started TM process and log them according to the user logging setup line by line
 * if BashJavaUtils fails then use default JVM args to start JVM process and log the failure

We could just disable logs in the run of BashJavaUtils and call again BashJavaUtils in the beginning of main standalone TM process to log but we would like to avoid duplicated calls as the computation can get diverged easily, like some later changes to Flink config in between etc.

> Preserve logs from BashJavaUtils and make them part of TM logs
> --------------------------------------------------------------
>
>                 Key: FLINK-15519
>                 URL: https://issues.apache.org/jira/browse/FLINK-15519
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Configuration
>    Affects Versions: 1.10.0
>            Reporter: Andrey Zagrebin
>            Priority: Blocker
>             Fix For: 1.10.0
>
>
> In FLINK-13983 we introduced BashJavaUtils utility to call in taskmanager.sh before starting TM and calculate memory configuration for the JVM process of TM.
> Ideally, it would be nice to preserve BashJavaUtils logs and make them part of the TM logs. Currently, logging for BashJavaUtils is configured from the class path and can differ from TM logging. Moreover TM logging can rewrite BashJavaUtils even if we align their loggings (e.g. log4j.appender.file.append=false in default log4j.properties  for Flink).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)