You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by zh...@apache.org on 2025/06/17 03:01:35 UTC
(pulsar) branch fix_style_log created (now 9638e3f8ba5)
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a change to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
at 9638e3f8ba5 fix style log
This branch includes the following new commits:
new db0136c82b6 update pom scm info
new 5b312afdedc close PerfDisableSharedMem
new a015f03c5e3 Merge branch 'close_PerfDisableSharedMem' into 'kwai-3.3.6'
new e5296e6ffc3 startup log && async logger
new a753cbcd0b7 set jute.maxbuffer to 30MB, as the meta data size of 240k partition is larger than 10M
new 9638e3f8ba5 fix style log
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(pulsar) 02/06: close PerfDisableSharedMem
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a commit to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 5b312afdedc8a1e9d2f29913f92ff6b08347960f
Author: fangxiaobing <fa...@kuaishou.com>
AuthorDate: Tue May 20 09:36:04 2025 +0000
close PerfDisableSharedMem
KDev_MR_link：https://ksurl.cn/ivGJZ3zY
---
conf/pulsar_env.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/conf/pulsar_env.sh b/conf/pulsar_env.sh
index f95d0ac83c1..bb10ae995b0 100755
--- a/conf/pulsar_env.sh
+++ b/conf/pulsar_env.sh
@@ -66,7 +66,7 @@ done
# Garbage collection options
if [ -z "$PULSAR_GC" ]; then
- PULSAR_GC="-XX:+PerfDisableSharedMem -XX:+AlwaysPreTouch"
+ PULSAR_GC="-XX:+AlwaysPreTouch"
if [[ $JAVA_MAJOR_VERSION -ge 21 ]]; then
PULSAR_GC="-XX:+UseZGC -XX:+ZGenerational ${PULSAR_GC}"
else
(pulsar) 05/06: set jute.maxbuffer to 30MB, as the meta data size of 240k partition is larger than 10M
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a commit to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a753cbcd0b781ddb09ed62aeffced2b6425759ca
Author: suiyuzeng <su...@kuaishou.com>
AuthorDate: Fri Jun 6 08:17:38 2025 +0000
set jute.maxbuffer to 30MB, as the meta data size of 240k partition is larger than 10M
KDev_MR_link：https://ksurl.cn/JZ4BcCmT
---
bin/pulsar | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/pulsar b/bin/pulsar
index 7b5ee6c454d..ecda7cf7979 100755
--- a/bin/pulsar
+++ b/bin/pulsar
@@ -250,7 +250,7 @@ OPTS="$OPTS -Dlog4j.configurationFile=`basename $PULSAR_LOG_CONF`"
# Ensure we can read bigger content from ZK. (It might be
# rarely needed when trying to list many z-nodes under a
# directory)
-OPTS="$OPTS -Djute.maxbuffer=10485760 -Djava.net.preferIPv4Stack=true"
+OPTS="$OPTS -Djute.maxbuffer=31457280 -Djava.net.preferIPv4Stack=true"
# Enable TCP keepalive for all Zookeeper client connections
OPTS="$OPTS -Dzookeeper.clientTcpKeepAlive=true"
(pulsar) 01/06: update pom scm info
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a commit to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit db0136c82b67784fe3bac43d652404a6bbc5a5bf
Author: zhaizhibo <zh...@kuaishou.com>
AuthorDate: Thu Jul 11 17:17:53 2024 +0800
update pom scm info
---
pom.xml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/pom.xml b/pom.xml
index a41c9b29591..3571fdf9a33 100644
--- a/pom.xml
+++ b/pom.xml
@@ -61,9 +61,10 @@ flexible messaging model and an intuitive client API.</description>
</licenses>
<scm>
- <url>https://github.com/apache/pulsar</url>
- <connection>scm:git:https://github.com/apache/pulsar.git</connection>
- <developerConnection>scm:git:ssh://git@github.com:apache/pulsar.git</developerConnection>
+ <url>https://git.corp.kuaishou.com/infra/pulsar</url>
+ <connection>scm:git:https://git.corp.kuaishou.com:infra/pulsar.git</connection>
+ <developerConnection>scm:git:ssh://git.corp.kuaishou.com:infra/pulsar.git</developerConnection>
+ <tag>HEAD</tag>
</scm>
<ciManagement>
(pulsar) 06/06: fix style log
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a commit to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9638e3f8ba5c2fdfb110af15d7e1b3a014a4bdd4
Author: gaozhangmin <gaozhangmin>
AuthorDate: Fri Jun 13 11:53:22 2025 +0800
fix style log
---
build/run_unit_group.sh | 2 +
conf/log4j2.yaml | 221 +++++++++++----------
.../java/org/apache/pulsar/utils/StartUpUtil.java | 8 +-
3 files changed, 124 insertions(+), 107 deletions(-)
diff --git a/build/run_unit_group.sh b/build/run_unit_group.sh
index 40f83efaf2f..785a9113063 100755
--- a/build/run_unit_group.sh
+++ b/build/run_unit_group.sh
@@ -177,6 +177,8 @@ function test_group_other() {
**/OffloadersCacheTest.java
**/PrimitiveSchemaTest.java,
**/BlobStoreManagedLedgerOffloaderTest.java,
+ **/FileSystemManagedLedgerOffloaderTest.java,
+ **/FileSystemOffloaderLocalFileTest.java,
**/BlobStoreManagedLedgerOffloaderStreamingTest.java'
mvn_test -pl managed-ledger -Dinclude='**/ManagedLedgerTest.java,
diff --git a/conf/log4j2.yaml b/conf/log4j2.yaml
index 0e49503581c..535f650055b 100644
--- a/conf/log4j2.yaml
+++ b/conf/log4j2.yaml
@@ -21,128 +21,148 @@
Configuration:
status: ERROR
monitorInterval: 30
- name: pulsar
+ name: kop
packages: io.prometheus.client.log4j2
Properties:
Property:
- - name: "pulsar.log.dir"
- value: "logs"
- - name: "pulsar.log.file"
- value: "pulsar.log"
- - name: "pulsar.log.immediateFlush"
- value: "false"
- - name: "pulsar.log.appender"
- value: "RoutingAppender"
- - name: "pulsar.log.root.level"
+ - name: "broker.log.dir"
+ value: "/home/web_server/kuaishou-runner/log"
+ - name: "broker.log.file"
+ value: "main.log"
+ - name: "kop.log.file"
+ value: "kop.log"
+ - name: "startup.log.file"
+ value: "startup.log"
+ - name: "request.log.file"
+ value: "request.log"
+ - name: "broker.log.appender"
+ value: "RollingFile"
+ - name: "broker.log.root.level"
value: "info"
- - name: "pulsar.log.level"
+ - name: "broker.log.level"
+ value: "info"
+ - name: "kop.log.level"
value: "info"
- - name: "pulsar.routing.appender.default"
- value: "Console"
-
- # Example: logger-filter script
-# Scripts:
-# ScriptFile:
-# name: filter.js
-# language: JavaScript
-# path: ./conf/log4j2-scripts/filter.js
-# charset: UTF-8
Appenders:
-
- # Console
Console:
name: Console
target: SYSTEM_OUT
PatternLayout:
Pattern: "%d{ISO8601_OFFSET_DATE_TIME_HHMM} [%t] %-5level %logger{36} - %msg%n"
- # Rolling file appender configuration
RollingFile:
- name: RollingFile
- fileName: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}"
- filePattern: "${sys:pulsar.log.dir}/${sys:pulsar.log.file}-%d{MM-dd-yyyy}-%i.log.gz"
- immediateFlush: ${sys:pulsar.log.immediateFlush}
- PatternLayout:
- Pattern: "%d{ISO8601_OFFSET_DATE_TIME_HHMM} [%t] %-5level %logger{36} - %msg%n"
- Policies:
- TimeBasedTriggeringPolicy:
- interval: 1
- modulate: true
- SizeBasedTriggeringPolicy:
- size: 1 GB
- # Delete file older than 30days
- DefaultRolloverStrategy:
+ - name: RollingFile
+ fileName: "${sys:broker.log.dir}/${sys:broker.log.file}"
+ filePattern: "${sys:broker.log.dir}/${sys:broker.log.file}-%d{MM-dd-yyyy}-%i.log.gz"
+ immediateFlush: true
+ PatternLayout:
+ Pattern: "%d{ISO8601_OFFSET_DATE_TIME_HHMM} [%t] %highlight{%-5level}{ERROR=red, WARN=yellow, INFO=green, DEBUG=blue, TRACE=magenta} %logger{36}.%method(%F:%L) - %highlight{%msg}%n"
+ Policies:
+ TimeBasedTriggeringPolicy:
+ interval: 1
+ modulate: true
+ SizeBasedTriggeringPolicy:
+ size: 1 GB
+ # Delete file older than 30days
+ DefaultRolloverStrategy:
+ Delete:
+ basePath: "${sys:broker.log.dir}"
+ maxDepth: 2
+ IfFileName:
+ glob: "${sys:broker.log.file}*log.gz"
+ IfLastModified:
+ age: 30d
+
+ - name: SERVICE_START_UP_LOGGER_APPENDER
+ fileName: "${sys:broker.log.dir}/${sys:startup.log.file}"
+ filePattern: "${sys:broker.log.dir}/${sys:startup.log.file}-%d{MM-dd-yyyy}-%i.log.gz"
+ immediateFlush: true
+ PatternLayout:
+ Pattern: "%msg%n"
+ Policies:
+ TimeBasedTriggeringPolicy:
+ interval: 1
+ modulate: true
+ SizeBasedTriggeringPolicy:
+ size: 1 GB
+ DefaultRolloverStrategy:
Delete:
- basePath: ${sys:pulsar.log.dir}
+ basePath: "${sys:broker.log.dir}"
maxDepth: 2
IfFileName:
- glob: "${sys:pulsar.log.file}*log.gz"
+ glob: "${sys:startup.log.file}*log.gz"
+ IfLastModified:
+ age: 30d
+
+ - name: KOP_LOGGER_APPENDER
+ fileName: "${sys:broker.log.dir}/${sys:kop.log.file}"
+ filePattern: "${sys:broker.log.dir}/${sys:kop.log.file}-%d{MM-dd-yyyy}-%i.log.gz"
+ immediateFlush: true
+ PatternLayout:
+ Pattern: "%d{ISO8601_OFFSET_DATE_TIME_HHMM} [%t] %highlight{%-5level}{ERROR=red, WARN=yellow, INFO=green, DEBUG=blue, TRACE=magenta} %logger{36}.%method(%F:%L) - %highlight{%msg}%n"
+ Policies:
+ TimeBasedTriggeringPolicy:
+ interval: 1
+ modulate: true
+ SizeBasedTriggeringPolicy:
+ size: 1 GB
+ DefaultRolloverStrategy:
+ Delete:
+ basePath: "${sys:broker.log.dir}"
+ maxDepth: 2
+ IfFileName:
+ glob: "${sys:kop.log.file}*log.gz"
+ IfLastModified:
+ age: 30d
+
+ - name: REQUEST_LOGGER_APPENDER
+ fileName: "${sys:broker.log.dir}/${sys:request.log.file}"
+ filePattern: "${sys:broker.log.dir}/${sys:request.log.file}-%d{MM-dd-yyyy}-%i.log.gz"
+ immediateFlush: true
+ PatternLayout:
+ Pattern: "%d{ISO8601_OFFSET_DATE_TIME_HHMM} [%t] %highlight{%-5level}{ERROR=red, WARN=yellow, INFO=green, DEBUG=blue, TRACE=magenta} %logger{36}.%method(%F:%L) - %highlight{%msg}%n"
+ Policies:
+ TimeBasedTriggeringPolicy:
+ interval: 1
+ modulate: true
+ SizeBasedTriggeringPolicy:
+ size: 1 GB
+ DefaultRolloverStrategy:
+ Delete:
+ basePath: "${sys:broker.log.dir}"
+ maxDepth: 2
+ IfFileName:
+ glob: "${sys:request.log.file}*log.gz"
IfLastModified:
age: 30d
Prometheus:
name: Prometheus
- # Routing
- Routing:
- name: RoutingAppender
- Routes:
- pattern: "$${ctx:function}"
- Route:
- -
- Routing:
- name: InstanceRoutingAppender
- Routes:
- pattern: "$${ctx:instance}"
- Route:
- -
- RollingFile:
- name: "Rolling-${ctx:function}"
- fileName : "${sys:pulsar.log.dir}/functions/${ctx:function}/${ctx:functionname}-${ctx:instance}.log"
- filePattern : "${sys:pulsar.log.dir}/functions/${sys:pulsar.log.file}-${ctx:instance}-%d{MM-dd-yyyy}-%i.log.gz"
- PatternLayout:
- Pattern: "%d{ABSOLUTE} %level{length=5} [%thread] [instance: %X{instance}] %logger{1} - %msg%n"
- Policies:
- TimeBasedTriggeringPolicy:
- interval: 1
- modulate: true
- SizeBasedTriggeringPolicy:
- size: "20MB"
- # Trigger every day at midnight that also scan
- # roll-over strategy that deletes older file
- CronTriggeringPolicy:
- schedule: "0 0 0 * * ?"
- # Delete file older than 30days
- DefaultRolloverStrategy:
- Delete:
- basePath: ${sys:pulsar.log.dir}
- maxDepth: 2
- IfFileName:
- glob: "${sys:pulsar.log.file}*log.gz"
- IfLastModified:
- age: 30d
- - ref: "${sys:pulsar.routing.appender.default}"
- key: "${ctx:function}"
- - ref: "${sys:pulsar.routing.appender.default}"
- key: "${ctx:function}"
-
Loggers:
-
- # Default root logger configuration
+ # Root logger configuration
Root:
- level: "${sys:pulsar.log.root.level}"
- additivity: true
+ level: "${sys:broker.log.root.level}"
+ includeLocation: true
AppenderRef:
- - ref: "${sys:pulsar.log.appender}"
- level: "${sys:pulsar.log.level}"
+ - ref: "${sys:broker.log.appender}"
+ level: "${sys:broker.log.level}"
- ref: Prometheus
level: info
Logger:
+ - name: io.streamnative.pulsar.handlers.kop
+ level: "${sys:kop.log.level}"
+ includeLocation: true
+ additivity: false
+ AppenderRef:
+ - ref: KOP_LOGGER_APPENDER
+
- name: org.apache.bookkeeper.bookie.BookieShell
level: info
+ includeLocation: true
additivity: false
AppenderRef:
- ref: Console
@@ -153,14 +173,15 @@ Configuration:
AppenderRef:
- ref: Console
- # Logger to inject filter script
-# - name: org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl
-# level: debug
-# additivity: false
-# AppenderRef:
-# ref: "${sys:pulsar.log.appender}"
-# ScriptFilter:
-# onMatch: ACCEPT
-# onMisMatch: DENY
-# ScriptRef:
-# ref: filter.js
+ - name: org.apache.pulsar.utils.StartUpUtil
+ level: info
+ additivity: false
+ AppenderRef:
+ - ref: SERVICE_START_UP_LOGGER_APPENDER
+
+ - name: org.eclipse.jetty.server.RequestLog
+ level: info
+ includeLocation: true
+ additivity: false
+ AppenderRef:
+ - ref: REQUEST_LOGGER_APPENDER
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java b/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java
index 3a841fa9298..7e839576808 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java
@@ -18,16 +18,10 @@
*/
package org.apache.pulsar.utils;
-import lombok.extern.slf4j.Slf4j;
-
import static java.lang.Integer.parseInt;
import static java.lang.management.ManagementFactory.getRuntimeMXBean;
+import lombok.extern.slf4j.Slf4j;
-
-/**
- * @author dongzhonghua03 <do...@kuaishou.com>
- * Created on 2025-05-19
- */
@Slf4j
public class StartUpUtil {
(pulsar) 04/06: startup log && async logger
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a commit to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit e5296e6ffc318e199a07288cc03b15f66d383c5f
Author: dongzhonghua03 <do...@kuaishou.com>
AuthorDate: Wed May 21 06:49:09 2025 +0000
startup log && async logger
KDev_MR_link：https://ksurl.cn/Y6jJ-AIs
---
pulsar-broker/pom.xml | 4 ++
.../org/apache/pulsar/PulsarBrokerStarter.java | 4 ++
.../java/org/apache/pulsar/utils/StartUpUtil.java | 57 ++++++++++++++++++++++
3 files changed, 65 insertions(+)
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index 84afc89fe82..3953324e726 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -501,6 +501,10 @@
<artifactId>pulsar-package-filesystem-storage</artifactId>
<version>${project.version}</version>
</dependency>
+ <dependency>
+ <groupId>com.lmax</groupId>
+ <artifactId>disruptor</artifactId>
+ </dependency>
</dependencies>
<build>
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java
index 2d031cc8a74..f6c65389c0e 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarBrokerStarter.java
@@ -58,6 +58,7 @@ import org.apache.pulsar.docs.tools.CmdGenerateDocs;
import org.apache.pulsar.functions.worker.WorkerConfig;
import org.apache.pulsar.functions.worker.WorkerService;
import org.apache.pulsar.functions.worker.service.WorkerServiceLoader;
+import org.apache.pulsar.utils.StartUpUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import picocli.CommandLine;
@@ -365,10 +366,13 @@ public class PulsarBrokerStarter {
try {
int start = starter.start(args);
if (start != 0) {
+ StartUpUtil.logServiceStartUp(false);
System.exit(start);
}
+ StartUpUtil.logServiceStartUp(true);
} catch (Throwable t) {
log.error("Failed to start pulsar service.", t);
+ StartUpUtil.logServiceStartUp(false);
ShutdownUtil.triggerImmediateForcefulShutdown();
} finally {
starter.join();
diff --git a/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java b/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java
new file mode 100644
index 00000000000..3a841fa9298
--- /dev/null
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/utils/StartUpUtil.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.utils;
+
+import lombok.extern.slf4j.Slf4j;
+
+import static java.lang.Integer.parseInt;
+import static java.lang.management.ManagementFactory.getRuntimeMXBean;
+
+
+/**
+ * @author dongzhonghua03 <do...@kuaishou.com>
+ * Created on 2025-05-19
+ */
+@Slf4j
+public class StartUpUtil {
+
+ private static volatile Integer pid;
+ private static final int UNKNOWN_PID = -1;
+
+ public static void logServiceStartUp(boolean success) {
+ log.info("[SERVICE_START_UP] pid={},status={},start_time={}", getPid(),
+ success ? "success" : "error", System.currentTimeMillis());
+ }
+
+ public static int getPid() {
+ if (pid == null) {
+ synchronized (StartUpUtil.class) {
+ if (pid == null) {
+ try {
+ String name = getRuntimeMXBean().getName();
+ pid = parseInt(name.substring(0, name.indexOf('@')));
+ } catch (Throwable e) {
+ pid = UNKNOWN_PID;
+ }
+ }
+ }
+ }
+ return pid;
+ }
+}
(pulsar) 03/06: Merge branch 'close_PerfDisableSharedMem' into 'kwai-3.3.6'
Posted by zh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zhangmingao pushed a commit to branch fix_style_log
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit a015f03c5e3bd61dce29a99380de29726341e598
Merge: db0136c82b6 5b312afdedc
Author: suiyuzeng <su...@kuaishou.com>
AuthorDate: Tue May 20 09:36:04 2025 +0000
Merge branch 'close_PerfDisableSharedMem' into 'kwai-3.3.6'
close PerfDisableSharedMem
See merge request pulsar!1!
KDev_MR_link：https://ksurl.cn/ivGJZ3zY
conf/pulsar_env.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)