You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "huwh (via GitHub)" <gi...@apache.org> on 2023/03/09 13:08:13 UTC
[GitHub] [flink] huwh opened a new pull request, #22138: [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir.
huwh opened a new pull request, #22138:
URL: https://github.com/apache/flink/pull/22138
## What is the purpose of the change
Add an option for redirecting stdout/stderr to files. When this is enabled.
1. flink-console.sh will redirect stdout/err to file.
2. flink-console.sh use log4j.properties as log4j configuration to avoid logs both in log file and stdout file.
Of course, this option is false by default.
## Brief change log
- *introduce new option kubernetes.flink.std.redirect.enabled, default is false*
- *when enabled, use log4j.properties/logback.xml as log configuration to avoid output log duplicated*
- *when enabled, redirect stdout/stderr to log directory so that user can view these in Flink's web ui*
## Verifying this change
- *Added unit tests for FlinkConfMountDecoratorTest.java*
- *Manually verified the change with kubernetes.flink.std.redirect.enabled is true or false.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22138: [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir.
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22138:
URL: https://github.com/apache/flink/pull/22138#issuecomment-1462045404
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7b2c0e82f4e1eec411ec6d870cd0e23cf4109470",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7b2c0e82f4e1eec411ec6d870cd0e23cf4109470",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7b2c0e82f4e1eec411ec6d870cd0e23cf4109470 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] huwh commented on pull request #22138: [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir.
Posted by "huwh (via GitHub)" <gi...@apache.org>.
huwh commented on PR #22138:
URL: https://github.com/apache/flink/pull/22138#issuecomment-1611108492
@wangyang0918 , Thanks for your review. comments addressed. And verified by running on Kubernetes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a diff in pull request #22138: [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir.
Posted by "wangyang0918 (via GitHub)" <gi...@apache.org>.
wangyang0918 commented on code in PR #22138:
URL: https://github.com/apache/flink/pull/22138#discussion_r1244777238
##########
flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java:
##########
@@ -307,6 +307,20 @@ public static String[] mergeListsToArray(List<String> base, List<String> append)
.defaultValue(10)
.withDescription("The maximum number of old log files to keep.");
+ /**
+ * This option is here only for documentation generation, it is only evaluated in the shell
+ * scripts.
+ */
+ @SuppressWarnings("unused")
+ public static final ConfigOption<Boolean> FLINK_STD_REDIRECT =
+ ConfigOptions.key("env.std.redirect")
Review Comment:
Maybe the key name could be `env.stdout-err.redirect-to-file`. It is more explicit.
##########
flink-dist/src/main/flink-bin/bin/flink-console.sh:
##########
@@ -116,4 +118,11 @@ echo $$ >> "$pid" 2>/dev/null
# Evaluate user options for local variable expansion
FLINK_ENV_JAVA_OPTS=$(eval echo ${FLINK_ENV_JAVA_OPTS})
+if [ "${STD_REDIRECT}" == "true" ]; then
+ # disable console appender to avoid redundant logs in log file and out file
Review Comment:
We never have redundant logs in log file. Right?
Here is only to avoid the redundant logs in out file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. [flink]
Posted by "xiaows08 (via GitHub)" <gi...@apache.org>.
xiaows08 commented on PR #22138:
URL: https://github.com/apache/flink/pull/22138#issuecomment-2401639724
@huwh 感谢回复,环境如下:
flink-kubernetes-operator 1.9.0
flink: flink:1.18.1-scala_2.12
log4j-console.properties 是镜像默认的
```yaml
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: basic
spec:
image: flink:1.18.1-scala_2.12
flinkVersion: v1_18
ingress:
className: "nginx"
template: "{{name}}.fo.xyuqing.com" # 引号必须有
podTemplate:
spec:
containers:
- name: flink-main-container
ports:
- name: metrics
containerPort: 9249
protocol: TCP
env:
- name: TZ # 设置容器运行的时区
value: Asia/Shanghai
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
env.stdout-err.redirect-to-file: "true"
metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
serviceAccount: flink
jobManager:
resource:
cpu: 1
memory: 2G
taskManager:
resource:
cpu: 1
memory: 2G
job:
jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
parallelism: 2
upgradeMode: stateless
```


我的意思是虽然 `kubectl logs -f pods/xxx ` 不会打印log4j2的日志内容,但是对应的 `xxx.out` 文件还是包含有 log4j2 `xxx.log`的日志内容,最终导致 代码中的log4j的输出会存在两份,也就会占两份存储,而且`xxx.out` 文件不能做到按大小或时间自动切分
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir. [flink]
Posted by "huwh (via GitHub)" <gi...@apache.org>.
huwh commented on PR #22138:
URL: https://github.com/apache/flink/pull/22138#issuecomment-2408448977
> @huwh 感谢回复,环境如下: flink-kubernetes-operator 1.9.0 flink: flink:1.18.1-scala_2.12 log4j-console.properties 是镜像默认的
>
> ```yaml
> apiVersion: flink.apache.org/v1beta1
> kind: FlinkDeployment
> metadata:
> name: basic
> spec:
> image: flink:1.18.1-scala_2.12
> flinkVersion: v1_18
> ingress:
> className: "nginx"
> template: "{{name}}.fo.xyuqing.com" # 引号必须有
> podTemplate:
> spec:
> containers:
> - name: flink-main-container
> ports:
> - name: metrics
> containerPort: 9249
> protocol: TCP
> env:
> - name: TZ # 设置容器运行的时区
> value: Asia/Shanghai
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> env.stdout-err.redirect-to-file: "true"
> metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
> serviceAccount: flink
> jobManager:
> resource:
> cpu: 1
> memory: 2G
> taskManager:
> resource:
> cpu: 1
> memory: 2G
> job:
> jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
> parallelism: 2
> upgradeMode: stateless
> ```
>
> 
>
> 
>
> 我的意思是虽然 `kubectl logs -f pods/xxx ` 不会打印log4j2的日志内容,但是对应的 `xxx.out` 文件还是包含有 log4j2 `xxx.log`的日志内容,最终导致 代码中的log4j的输出会存在两份,也就会占两份存储,而且`xxx.out` 文件不能做到按大小或时间自动切分
This is because flink-kubernetes-operator uses its log4j-console.properties, which does not support the console appender filtering function.
Currently, flink-kubernetes-operator does not recommend enable env.stdout-err.redirect-to-file, which does cause log duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] wangyang0918 merged pull request #22138: [FLINK-31234][kubernetes] Add an option to redirect stdout to log dir.
Posted by "wangyang0918 (via GitHub)" <gi...@apache.org>.
wangyang0918 merged PR #22138:
URL: https://github.com/apache/flink/pull/22138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org