You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "SaketaChalamchala (via GitHub)" <gi...@apache.org> on 2023/02/13 17:18:10 UTC
[GitHub] [ozone] SaketaChalamchala opened a new pull request, #4271: HDDS-7869. Log configuration on component startup.
SaketaChalamchala opened a new pull request, #4271:
URL: https://github.com/apache/ozone/pull/4271
## What changes were proposed in this pull request?
Added a `conf` field in `STARTUP_MSG` for all Ozone roles with configs relevant to the role.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7869
## How was this patch tested?
Manual test
```
OM :
/************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting OzoneManager
...
2023-02-12 20:11:34 STARTUP_MSG: java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG: conf = {hadoop.hdds.db.rocksdb.logging.level=INFO, ozone.om.enable.ofs.shared.tmp.dir=false, ozone.om.ratis.log.appender.queue.num-elements=1024, hadoop.hdds.db.rocksdb.logging.enabled=false, ozone.om.snapshot.provider.socket.timeout=5000s, ozone.scm.info.wait.duration=10m, ozone.om.http.auth.type=kerberos, ozone.om.multitenancy.enabled=true, ozone.om.https-bind-host=0.0.0.0, ozone.om.ratis.server.request.timeout=3s, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.om.snapshot.compaction.dag.max.time.allowed=30d, ozone.path.deleting.limit.per.task=10000, ozone.om.enable.filesystem.paths=false, ozone.recon.om.snapshot.task.flush.param=false, ozone.om.admin.protocol.max.retries=20, ozone.om.leader.election.minimum.timeout.duration=5s, ozone.om.ratis.log.appender.queue.byte-limit=32MB, ozone.network.jvm.address.cache.enabled=true, ozone.om.volume.listall.allowed=true, ozone.recon.om.socket.timeout=5s, ozone.om.keyname.character.check.enab
led=false, ozone.om.open.key.cleanup.service.timeout=300s, ozone.om.ratis.server.retry.cache.timeout=600000ms, ozone.om.snapshot.provider.request.timeout=5000ms, ozone.directory.deleting.service.interval=1m, ozone.om.save.metrics.interval=5m, ozone.om.grpc.maximum.response.length=134217728, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.om.https-address=0.0.0.0:9875, ozone.om.ratis.port=9872, ozone.key.deleting.limit.per.task=20000, ozone.om.ha.raft.server.retrycache.expirytime=300s, ozone.snapshot.filtering.service.interval=1m, ozone.scm.ca.list.retry.interval=10s, ozone.om.container.location.cache.size=100000, ozone.om.handler.count.key=100, ozone.om.client.trash.core.pool.size=5, hdds.db.profile=DISK, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.fs.listing.page.size.max=5000, ozone.key.preallocation.max.blocks=64, ozone.om.http-bind-host=0.0.0.0, hdds.scm.http.auth.type=kerberos, ozone.om.fs.snapshot.max.limit=1000, ozone.om.snapshot.provider.connection.timeou
t=5000s, ozone.metadata.dirs=/data/metadata, ozone.recon.om.connection.timeout=5s, ozone.om.key.path.lock.enabled=false, ozone.metastore.rocksdb.statistics=OFF, ozone.om.ratis.log.purge.gap=1000000, ozone.om.group.rights=ALL, ozone.om.ratis.segment.size=4MB, ozone.om.multitenancy.ranger.sync.timeout=10s, ozone.om.open.key.expire.threshold=7d, ozone.om.snapshot.cache.max.size=10, ozone.om.user.max.volume=1024, ozone.om.client.rpc.timeout=15m, ozone.snapshot.filtering.limit.per.task=2, ozone.sst.filtering.service.timeout=300000ms, ozone.om.ratis.segment.preallocated.size=4MB, ozone.om.ratis.server.failure.timeout.duration=120s, ozone.om.unflushed.transaction.max.count=10000, ozone.om.user.rights=ALL, ozone.recon.om.snapshot.task.interval.delay=1m, ozone.om.container.location.cache.ttl=360m, ozone.om.upgrade.finalization.ratis.based.timeout=30s, ozone.om.http-address=om:9874, ozone.om.admin.protocol.wait.between.retries=1000, ozone.om.address=om, ozone.om.ratis.minimum.timeout=5s, ozon
e.om.snapshot.compaction.dag.prune.daemon.run.interval=3600s, ozone.om.open.key.cleanup.limit.per.task=1000, ozone.om.ratis.enable=true, ozone.service.shutdown.timeout=60s, hadoop.hdds.db.rocksdb.writeoption.sync=false, ozone.recon.om.connection.request.timeout=5000, ozone.om.multitenancy.ranger.sync.interval=30s, ozone.om.ratis.rpc.type=GRPC, ozone.om.ratis.server.leaderelection.pre-vote=false, ozone.om.http.enabled=true, ozone.om.open.key.cleanup.service.interval=24h, ozone.om.init.default.layout.version=-1}
```
```
SCM :
2023-02-12 20:11:34 /************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting StorageContainerManager
...
2023-02-12 20:11:34 STARTUP_MSG: java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG: conf = {ozone.scm.sequence.id.batch.size=1000, hdds.scm.replication.event.timeout=10s, ozone.scm.ha.ratis.snapshot.threshold=1000, hdds.scm.safemode.atleast.one.node.reported.pipeline.pct=0.90, ozone.scm.ha.ratis.server.leaderelection.pre-vote=false, hdds.scm.safemode.enabled=true, ozone.scm.client.port=9860, hdds.key.len=2048, ozone.network.jvm.address.cache.enabled=true, ozone.scm.pipeline.creation.interval=30s, hdds.datanode.storage.utilization.critical.threshold=0.95, hdds.scm.safemode.min.datanode=3, ozone.scm.client.bind.host=0.0.0.0, hdds.scm.block.deletion.per-interval.max=100000, hdds.scm.kerberos.keytab.file=/etc/security/keytabs/scm.keytab, ozone.scm.block.client.port=9863, ozone.scm.ha.ratis.segment.preallocated.size=4MB, hdds.scm.http.auth.kerberos.principal=HTTP/scm@EXAMPLE.COM, ozone.scm.block.deletion.max.retry=4096, ozone.scm.ratis.pipeline.limit=0, ozone.block.deleting.container.limit.per.interval=10, hdds.scm.http.auth.kerberos
.keytab=/etc/security/keytabs/scm.keytab, ozone.block.deleting.service.interval=1m, ozone.scm.ca.list.retry.interval=10s, ozone.scm.ha.ratis.leader.election.timeout=5s, hdds.scm.safemode.pipeline.creation=true, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.chunk.read.buffer.default.size=64KB, ozone.scm.ha.ratis.log.appender.queue.byte-limit=32MB, hdds.scm.kerberos.principal=scm/scm@EXAMPLE.COM, hdds.scmclient.failover.retry.interval=2s, hdds.scm.replication.maintenance.remaining.redundancy=1, ozone.scm.container.layout=FILE_PER_BLOCK, ozone.block.deleting.limit.per.task=1000, hdds.scm.replication.container.inflight.replication.limit=0, hdds.scmclient.max.retry.timeout=30s, hdds.datanode.block.deleting.service.interval=60s, ozone.scm.expired.container.replica.op.scrub.interval=5m, ozone.scm.block.client.bind.host=0.0.0.0, hdds.datanode.recovering.container.scrubbing.service.interval=1m, ozone.scm.block.client.address=scm, ozone.scm.ha.grpc.deadline.interval=30m, hdds.scm.
init.default.layout.version=-1, hdds.scm.replication.command.deadline.factor=0.9, ozone.scm.pipeline.destroy.timeout=66s, ozone.recon.scm.snapshot.enabled=true, ozone.scm.pipeline.per.metadata.disk=2, hdds.datanode.storage.utilization.warning.threshold=0.75, hdds.key.dir.name=keys, ozone.scm.pipeline.owner.container.count=1, ozone.scm.ha.ratis.server.retry.cache.timeout=60s, ozone.scm.block.size=256MB, ozone.scm.pipeline.allocated.timeout=5m, hdds.scm.replication.thread.interval=5s, ozone.scm.ha.ratis.log.purge.enabled=false, ozone.scm.datanode.admin.monitor.interval=30s, ozone.scm.info.wait.duration=10m, hdds.scm.replication.maintenance.replica.minimum=2, hdds.scm.unknown-container.action=WARN, hdds.scm.replication.over.replicated.interval=30s, hdds.scmclient.failover.max.retry=15, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.scm.ha.ratis.request.timeout=30s, hdds.scm.safemode.threshold.pct=0.99, ozone.scm.ratis.port=9894, hdds.scm.safemode.healthy.pipeline.pct=0.10,
ozone.scm.ha.ratis.leader.ready.wait.timeout=60s, hdds.scm.wait.time.after.safemode.exit=5m, ozone.recon.scm.container.threshold=100, ozone.scm.pipeline.scrub.interval=5m, ozone.recon.scm.connection.request.timeout=5s, ozone.block.deleting.service.timeout=300000ms, ozone.block.deleting.service.workers=10, hdds.scm.block.deleting.service.interval=60s, ozone.scm.skip.bootstrap.validation=false, ozone.scm.ha.ratis.leader.ready.check.interval=2s, ozone.scm.datanode.disallow.same.peers=false, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.pipeline.creation.auto.factor.one=true, hdds.scm.safemode.pipeline-availability.check=true, ozone.scm.event.ContainerReport.thread.pool.size=10, hdds.scm.replication.container.inflight.deletion.limit=0, ozone.metadata.dirs=/data/metadata, ozone.scm.ha.ratis.log.appender.queue.num-elements=1024, ozone.scm.ha.ratis.segment.size=4MB, ozone.scm.ha.ratis.rpc.type=GRPC, ozone.metastore.rocksdb.statistics=OFF, hdds.scm.replication.push=true, ozone
.scm.datanode.pipeline.limit=1, ozone.scm.grpc.port=9895, hdds.scm.replication.under.replicated.interval=30s, ozone.recon.scm.connection.timeout=5s, hdds.scm.pipeline.choose.policy.impl=org.apache.hadoop.hdds.scm.pipeline.choose.algorithms.RandomPipelineChoosePolicy, ozone.scm.ha.ratis.server.failure.timeout.duration=120s, hdds.datanode.block.deleting.limit.per.interval=5000, ozone.service.shutdown.timeout=60s, ozone.scm.client.address=scm, ozone.scm.chunk.size=4MB, ozone.scm.ha.ratis.log.purge.gap=1000000, ozone.scm.pipeline.leader-choose.policy=
2023-02-12 20:11:34 org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
2023-02-12 20:11:34 , hdds.scmclient.rpc.timeout=15m}
2023-02-12 20:11:34 ************************************************************/
```
```
Recon:
/************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting ReconServer
...
2023-02-12 20:11:34 STARTUP_MSG: java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG: conf = {ozone.recon.http.auth.kerberos.principal=HTTP/recon@EXAMPLE.COM, ozone.recon.task.missingcontainer.interval=300s, ozone.recon.sql.db.conn.idle.max.age=3600s, ozone.recon.http.enabled=true, ozone.recon.task.pipelinesync.interval=300s, ozone.recon.http-address=0.0.0.0:9888, ozone.recon.nssummary.flush.db.max.threshold=150000, ozone.recon.om.connection.timeout=5s, recon.om.delta.update.loop.limit=10, ozone.recon.administrators=testuser2/scm@EXAMPLE.COM, recon.om.delta.update.limit=2000, ozone.recon.om.snapshot.task.flush.param=false, ozone.recon.sql.db.driver=org.apache.derby.jdbc.EmbeddedDriver, ozone.recon.security.client.datanode.container.protocol.acl=*, ozone.recon.scm.connection.timeout=5s, hdds.recon.heartbeat.interval=60s, ozone.recon.scm.container.threshold=100, ozone.recon.db.dir=/data/metadata/recon, ozone.recon.om.socket.timeout=5s, ozone.recon.sql.db.conn.timeout=30000ms, ozone.recon.https-bind-host=0.0.0.0, ozone.recon.om.snaps
hot.task.interval.delay=1m, ozone.recon.task.thread.count=1, ozone.recon.http.auth.type=kerberos, ozone.recon.sql.db.conn.idle.test.period=60s, ozone.recon.kerberos.principal=recon/recon@EXAMPLE.COM, ozone.recon.sql.db.conn.idle.test=SELECT 1, ozone.recon.sql.db.auto.commit=true, ozone.recon.scm.snapshot.enabled=true, ozone.recon.http-bind-host=0.0.0.0, ozone.recon.sql.db.jdbc.url=jdbc:derby:${ozone.recon.db.dir}/ozone_recon_derby.db, ozone.recon.scm.connection.request.timeout=5s, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.service.shutdown.timeout=60s, ozone.recon.sql.db.conn.max.active=5, ozone.recon.scm.snapshot.task.initial.delay=1m, ozone.recon.scm.snapshot.task.interval.delay=24h, ozone.recon.om.connection.request.timeout=5000, ozone.recon.http.auth.kerberos.keytab=/etc/security/keytabs/recon.keytab, ozone.recon.task.safemode.wait.threshold=300s, ozone.recon.sql.db.jooq.dialect=DERBY, ozone.recon.sql.db.conn.max.age=1800s, ozone.recon.kerberos.keytab.file=/etc/securi
ty/keytabs/recon.keytab, ozone.recon.https-address=0.0.0.0:9889}
```
```
DN:
/************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting HddsDatanodeService
...
2023-02-12 20:11:34 STARTUP_MSG: java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG: conf = {hdds.datanode.metadata.rocksdb.cache.size=64MB, ozone.scm.datanode.ratis.volume.free-space.min=1GB, hdds.datanode.rocksdb.auto-compaction-small-sst-file-num-threshold=512, hdds.ratis.raft.server.delete.ratis.log.directory=true, ozone.network.jvm.address.cache.enabled=true, hdds.datanode.chunk.data.validation.check=false, hdds.datanode.rocksdb.log.level=INFO, hdds.datanode.container.delete.threads.max=2, hdds.ratis.raft.server.watch.timeout=180s, hdds.datanode.container.schema.v3.enabled=true, hdds.datanode.block.delete.threads.max=5, hdds.ratis.raft.server.notification.no-leader.timeout=300s, hdds.ratis.raft.server.datastream.client.pool.size=10, hdds.ratis.raft.server.datastream.request.threads=20, ozone.fs.datastream.enabled=false, hdds.datanode.wait.on.all.followers=false, hdds.ratis.raft.server.rpc.slowness.timeout=300s, ozone.scm.ca.list.retry.interval=10s, hdds.datanode.replication.streams.limit=10, hdds.datanode.rocksdb.auto-compac
tion-small-sst-file=true, hdds.datanode.failed.data.volumes.tolerated=-1, hdds.datanode.ratis.server.request.timeout=2m, hdds.datanode.rocksdb.max-open-files=1024, hdds.datanode.rocksdb.auto-compaction-small-sst-file-size-threshold=1MB, hdds.datanode.read.chunk.threads.per.volume=10, hdds.datanode.rocksdb.delete-obsolete-files-period=1h, hdds.datanode.container.schema.v3.key.separator=|, hdds.command.status.report.interval=30s, hdds.ratis.raft.server.leaderelection.pre-vote=false, hdds.datanode.block.delete.queue.limit=1440, hdds.ratis.raft.server.rpc.request.timeout=60s, hdds.container.replication.compression=NO_COMPRESSION, hdds.container.action.max.limit=20, hdds.datanode.periodic.disk.check.interval.minutes=60, hdds.datanode.rocksdb.log.max-file-num=64, hdds.datanode.df.refresh.period=5m, hdds.pipeline.action.max.limit=20, hdds.datanode.failed.db.volumes.tolerated=-1, hdds.datanode.rocksdb.log.max-file-size=32MB, hdds.datanode.disk.check.timeout=10m, hdds.container.close.thresho
ld=0.9f, hdds.datanode.disk.check.min.gap=15m, ozone.service.shutdown.timeout=60s, hdds.ratis.raft.server.write.element-limit=1024, hdds.datanode.replication.port=9886, hdds.container.checksum.verification.enabled=true, hdds.datanode.failed.metadata.volumes.tolerated=-1, hdds.datanode.du.refresh.period=1h, hdds.datanode.http.auth.type=kerberos}
```
```
S3G:
/************************************************************
2023-02-12 17:03:21 STARTUP_MSG: Starting Gateway
...
2023-02-12 17:03:21 STARTUP_MSG: java = 11.0.17
2023-02-12 17:03:21 STARTUP_MSG: conf = {hadoop.http.idle_timeout.ms=60000, ozone.s3g.volume.name=s3v, ozone.s3g.http-address=0.0.0.0:9878, ozone.om.grpc.maximum.response.length=134217728, ozone.s3g.http.enabled=true, ozone.service.shutdown.timeout=60s, ozone.s3g.client.buffer.size=4KB, ozone.s3g.http.auth.kerberos.principal=HTTP/s3g@EXAMPLE.COM, ozone.s3g.kerberos.principal=s3g/s3g@EXAMPLE.COM, ozone.s3g.kerberos.keytab.file=/etc/security/keytabs/s3g.keytab, ozone.s3g.http.auth.kerberos.keytab=/etc/security/keytabs/s3g.keytab, ozone.s3g.http-bind-host=0.0.0.0}
```
```
Ozone CSI:
sh-4.2$ ozone csi -conf=/etc/hadoop/x --verbose
...
2023-02-13 01:05:19,080 [main] INFO csi.CsiServer: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting CsiServer
STARTUP_MSG: host = b453337cacf2/172.18.0.8
STARTUP_MSG: args = [-conf=/etc/hadoop/x, --verbose]
STARTUP_MSG: version = 1.4.0-SNAPSHOT
STARTUP_MSG: classpath = /etc/hadoop...
STARTUP_MSG: build = git@github.com:SaketaChalamchala/ozone.git/e2a52323ad6a6c0675717e89dd3e9f472e52c228 ; compiled by 'schalamchala' on 2023-02-13T02:12Z
STARTUP_MSG: java = 11.0.17
STARTUP_MSG: conf = {ozone.csi.s3g.address=http://localhost:9878, ozone.csi.mount.command=goofys --endpoint %s %s %s, ozone.csi.socket=/var/lib/csi.sock, ozone.csi.default-volume-size=1000000000}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1433868782
Thanks @errose28 @adoroszlai and @sumitagrawl for the review.
> I don't think we should filter by tags or prefixes when printing the config. I think we should just print the contents of whatever config object the component has.
I'm printing all the properties in the config object used by each service which includes core-site, ozone-site and all the defaults. Below are examples. It seems like a lot. Let me know if it's readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1119937597
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java:
##########
@@ -776,4 +779,22 @@ public static ContainerProtos.ContainerCommandResponseProto processForDebug(
return msg;
}
+
+ /**
+ * Redacts sensitive configuration.
+ * Sorts all properties by key name
+ *
+ * @param conf OzoneConfiguration object to be printed.
+ * @return Sorted Map of properties
+ */
+ public static Map<String, String> processForLogging(OzoneConfiguration conf) {
+ Map<String, String> ozoneProps = conf.getOzoneProperties();
+ ConfigRedactor redactor = new ConfigRedactor(conf);
+ Map<String, String> sortedOzoneProps = new TreeMap<>();
+ for (Map.Entry<String, String> entry : ozoneProps.entrySet()) {
+ String value = redactor.redact(entry.getKey(), entry.getValue());
+ sortedOzoneProps.put(entry.getKey(), value);
Review Comment:
This would take care of items with leading/trailing whitespace:
```suggestion
String value = redactor.redact(entry.getKey(), entry.getValue());
if (value != null) {
value = value.trim();
}
sortedOzoneProps.put(entry.getKey(), value);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1109119842
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java:
##########
@@ -105,12 +105,12 @@ public static String toStartupShutdownString(String prefix, String... msg) {
}
public static void startupShutdownMessage(VersionInfo versionInfo,
- Class<?> clazz, String[] args, Logger log) {
+ Class<?> clazz, String[] args, Logger log, String conf) {
Review Comment:
Passing the configuration object as it is and I added a `processForLogging` utility function to process the configuration object before logging it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1105783993
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java:
##########
@@ -105,12 +105,12 @@ public static String toStartupShutdownString(String prefix, String... msg) {
}
public static void startupShutdownMessage(VersionInfo versionInfo,
- Class<?> clazz, String[] args, Logger log) {
+ Class<?> clazz, String[] args, Logger log, String conf) {
Review Comment:
I think it would be better to pass a `Map` here instead of a fully formatted string. This would allow filtering some sensitive properties if needed, as well as sorting the config properties alphabetically. (Currently order is random, since `Properties` is just a `Hashtable`.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1105769925
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManagerStarter.java:
##########
@@ -156,7 +157,8 @@ private void commonInit() {
String[] originalArgs = getCmd().getParseResult().originalArgs()
.toArray(new String[0]);
StringUtils.startupShutdownMessage(HddsVersionInfo.HDDS_VERSION_INFO,
- StorageContainerManager.class, originalArgs, LOG);
+ StorageContainerManager.class, originalArgs, LOG,
+ conf.getAllPropertiesByTag(ConfigTag.SCM.name()).toString());
Review Comment:
I think fixing tags of specific config properties should be addressed in separate PR(s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1119627631
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java:
##########
@@ -776,4 +779,22 @@ public static ContainerProtos.ContainerCommandResponseProto processForDebug(
return msg;
}
+
+ /**
+ * Redacts sensitive configuration.
+ * Sorts all properties by key name
+ *
+ * @param conf OzoneConfiguration object to be printed.
+ * @return Sorted Map of properties
+ */
+ public static Map<String, String> processForLogging(OzoneConfiguration conf) {
+ Map<String, String> ozoneProps = conf.getOzoneProperties();
+ ConfigRedactor redactor = new ConfigRedactor(conf);
+ Map<String, String> sortedOzoneProps = new TreeMap<>();
+ for (String name : ozoneProps.keySet()) {
+ String value = redactor.redact(name, ozoneProps.get(name));
+ sortedOzoneProps.put(name, value);
Review Comment:
Findbugs complains:
```
M P WMI: org.apache.hadoop.hdds.HddsUtils.processForLogging(OzoneConfiguration) makes inefficient use of keySet iterator instead of entrySet iterator At HddsUtils.java:[line 795]
```
https://github.com/apache/ozone/actions/runs/4288734741/jobs/7471080023#step:6:2053
Can be fixed by:
```suggestion
for (Map.Entry<String, String> entry : ozoneProps.entrySet()) {
String value = redactor.redact(entry.getKey(), entry.getValue());
sortedOzoneProps.put(entry.getKey(), value);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1430591136
I don't think we should filter by tags or prefixes when printing the config. I think we should just print the contents of whatever config object the component has.
- I think this would show only options that the user has explicitly set, which may be different than the defaults. I think this set of configs would be more useful for debugging.
- We don't have to worry about missing config values in the log message due to improper tagging now or for future values.
- It would show an accurate representation of the config the component has, even if it does not read some of the values.
- This removes some assumptions about what components are supposed to read vs. what they could potentially read, which could hinder debugging.
- Since this would usually be inspected for debugging purposes, users can filter the configs they are looking for anyways.
Also +1 to Attila's suggestion of sorting the configs alphabetically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1447452717
Updated the patch to include Ozone configs only using regex.
```
## Full list of configs printed in all role logs :
STARTUP_MSG: conf = {dfs.container.chunk.write.sync=false, dfs.container.ipc=9859, dfs.container.ipc.random.port=false, dfs.container.ratis.admin.port=9857, dfs.container.ratis.datastream.enabled=true, dfs.container.ratis.datastream.port=9855, dfs.container.ratis.datastream.random.port=false, dfs.container.ratis.enabled=false, dfs.container.ratis.ipc=9858, dfs.container.ratis.ipc.random.port=false, dfs.container.ratis.leader.pending.bytes.limit=1GB, dfs.container.ratis.log.appender.queue.byte-limit=32MB, dfs.container.ratis.log.appender.queue.num-elements=1, dfs.container.ratis.log.purge.gap=1000000, dfs.container.ratis.log.queue.byte-limit=4GB, dfs.container.ratis.log.queue.num-elements=1024, dfs.container.ratis.num.container.op.executors=10, dfs.container.ratis.num.write.chunk.threads.per.volume=10, dfs.container.ratis.replication.level=MAJORITY, dfs.container.ratis.rpc.type=GRPC, dfs.container.ratis.segment.preallocated.size=16KB, dfs.container.ratis.segment.size=1MB, dfs.con
tainer.ratis.server.port=9856, dfs.container.ratis.statemachine.max.pending.apply-transactions=10000, dfs.container.ratis.statemachinedata.sync.retries=-1, dfs.container.ratis.statemachinedata.sync.timeout=10s, dfs.ratis.leader.election.minimum.timeout.duration=5s, dfs.ratis.server.retry-cache.timeout.duration=600000ms, dfs.ratis.snapshot.threshold=10000, fs.trash.classname=org.apache.hadoop.ozone.om.TrashPolicyOzone, hadoop.hdds.db.rocksdb.logging.enabled=false, hadoop.hdds.db.rocksdb.logging.level=INFO, hadoop.hdds.db.rocksdb.writeoption.sync=false, hadoop.http.idle_timeout.ms=60000, hdds.block.token.enabled=true, hdds.block.token.expiry.time=1d, hdds.command.status.report.interval=30s, hdds.container.action.max.limit=20, hdds.container.balancer.balancing.iteration.interval=70m, hdds.container.balancer.datanodes.involved.max.percentage.per.iteration=20, hdds.container.balancer.iterations=10, hdds.container.balancer.move.networkTopology.enable=false, hdds.container.balancer.move.ti
meout=30m, hdds.container.balancer.size.entering.target.max=26GB, hdds.container.balancer.size.leaving.source.max=26GB, hdds.container.balancer.size.moved.max.per.iteration=500GB, hdds.container.balancer.trigger.du.before.move.enable=false, hdds.container.balancer.utilization.threshold=10, hdds.container.checksum.verification.enabled=true, hdds.container.close.threshold=0.9f, hdds.container.replication.compression=NO_COMPRESSION, hdds.container.report.interval=60s, hdds.container.scrub.data.scan.interval=7d, hdds.container.scrub.enabled=false, hdds.container.scrub.metadata.scan.interval=3h, hdds.container.scrub.on.demand.volume.bytes.per.second=5242880, hdds.container.scrub.volume.bytes.per.second=5242880, hdds.container.token.enabled=true, hdds.crl.status.report.interval=60000ms, hdds.datanode.block.delete.queue.limit=1440, hdds.datanode.block.delete.threads.max=5, hdds.datanode.block.deleting.limit.per.interval=5000, hdds.datanode.block.deleting.service.interval=60s, hdds.datanode
.chunk.data.validation.check=false, hdds.datanode.container.delete.threads.max=2, hdds.datanode.container.schema.v3.enabled=true, hdds.datanode.container.schema.v3.key.separator=|, hdds.datanode.df.refresh.period=5m, hdds.datanode.dir=/data/hdds, hdds.datanode.disk.check.min.gap=15m, hdds.datanode.disk.check.timeout=10m, hdds.datanode.du.refresh.period=1h, hdds.datanode.failed.data.volumes.tolerated=-1, hdds.datanode.failed.db.volumes.tolerated=-1, hdds.datanode.failed.metadata.volumes.tolerated=-1, hdds.datanode.http-address=0.0.0.0:9882, hdds.datanode.http-bind-host=0.0.0.0, hdds.datanode.http.auth.kerberos.keytab=/etc/security/keytabs/dn.keytab, hdds.datanode.http.auth.kerberos.principal=HTTP/dn@EXAMPLE.COM, hdds.datanode.http.auth.type=kerberos, hdds.datanode.http.enabled=true, hdds.datanode.https-address=0.0.0.0:9883, hdds.datanode.https-bind-host=0.0.0.0, hdds.datanode.metadata.rocksdb.cache.size=64MB, hdds.datanode.periodic.disk.check.interval.minutes=60, hdds.datanode.ratis.
server.request.timeout=2m, hdds.datanode.read.chunk.threads.per.volume=10, hdds.datanode.recovering.container.scrubbing.service.interval=1m, hdds.datanode.replication.port=9886, hdds.datanode.replication.streams.limit=10, hdds.datanode.rocksdb.auto-compaction-small-sst-file=true, hdds.datanode.rocksdb.auto-compaction-small-sst-file-num-threshold=512, hdds.datanode.rocksdb.auto-compaction-small-sst-file-size-threshold=1MB, hdds.datanode.rocksdb.delete-obsolete-files-period=1h, hdds.datanode.rocksdb.log.level=INFO, hdds.datanode.rocksdb.log.max-file-num=64, hdds.datanode.rocksdb.log.max-file-size=32MB, hdds.datanode.rocksdb.max-open-files=1024, hdds.datanode.storage.utilization.critical.threshold=0.95, hdds.datanode.storage.utilization.warning.threshold=0.75, hdds.datanode.wait.on.all.followers=false, hdds.db.profile=DISK, hdds.grpc.tls.enabled=true, hdds.grpc.tls.provider=OPENSSL, hdds.heartbeat.interval=30s, hdds.key.dir.name=keys, hdds.key.len=2048, hdds.node.report.interval=60000m
s, hdds.pipeline.action.max.limit=20, hdds.pipeline.report.interval=60000ms, hdds.priv.key.file.name=private.pem, hdds.profiler.endpoint.enabled=false, hdds.prometheus.endpoint.enabled=true, hdds.public.key.file.name=public.pem, hdds.ratis.client.exponential.backoff.base.sleep=4s, hdds.ratis.client.exponential.backoff.max.sleep=40s, hdds.ratis.client.multilinear.random.retry.policy=5s, 5, 10s, 5, 15s, 5, 20s, 5, 25s, 5, 60s, 10, hdds.ratis.client.request.watch.timeout=3m, hdds.ratis.client.request.write.timeout=5m, hdds.ratis.client.retry.policy=org.apache.hadoop.hdds.ratis.retrypolicy.RequestTypeDependentRetryPolicyCreator, hdds.ratis.client.retrylimited.max.retries=180, hdds.ratis.client.retrylimited.retry.interval=1s, hdds.ratis.raft.client.async.outstanding-requests.max=32, hdds.ratis.raft.client.rpc.request.timeout=60s, hdds.ratis.raft.client.rpc.watch.request.timeout=180s, hdds.ratis.raft.grpc.flow.control.window=5MB, hdds.ratis.raft.grpc.message.size.max=32MB, hdds.ratis.raft
.server.datastream.client.pool.size=10, hdds.ratis.raft.server.datastream.request.threads=20, hdds.ratis.raft.server.delete.ratis.log.directory=true, hdds.ratis.raft.server.leaderelection.pre-vote=false, hdds.ratis.raft.server.notification.no-leader.timeout=300s, hdds.ratis.raft.server.rpc.request.timeout=60s, hdds.ratis.raft.server.rpc.slowness.timeout=300s, hdds.ratis.raft.server.watch.timeout=180s, hdds.ratis.raft.server.write.element-limit=1024, hdds.ratis.server.num.snapshots.retained=5, hdds.recon.heartbeat.interval=60s, hdds.scm.block.deleting.service.interval=60s, hdds.scm.block.deletion.per-interval.max=100000, hdds.scm.http.auth.kerberos.keytab=/etc/security/keytabs/scm.keytab, hdds.scm.http.auth.kerberos.principal=HTTP/scm@EXAMPLE.COM, hdds.scm.http.auth.type=kerberos, hdds.scm.init.default.layout.version=-1, hdds.scm.kerberos.keytab.file=/etc/security/keytabs/scm.keytab, hdds.scm.kerberos.principal=scm/scm@EXAMPLE.COM, hdds.scm.pipeline.choose.policy.impl=org.apache.hado
op.hdds.scm.pipeline.choose.algorithms.RandomPipelineChoosePolicy, hdds.scm.replication.command.deadline.factor=0.9, hdds.scm.replication.container.inflight.deletion.limit=0, hdds.scm.replication.container.inflight.replication.limit=0, hdds.scm.replication.event.timeout=10s, hdds.scm.replication.maintenance.remaining.redundancy=1, hdds.scm.replication.maintenance.replica.minimum=2, hdds.scm.replication.over.replicated.interval=30s, hdds.scm.replication.push=true, hdds.scm.replication.thread.interval=5s, hdds.scm.replication.under.replicated.interval=30s, hdds.scm.safemode.atleast.one.node.reported.pipeline.pct=0.90, hdds.scm.safemode.enabled=true, hdds.scm.safemode.healthy.pipeline.pct=0.10, hdds.scm.safemode.min.datanode=3, hdds.scm.safemode.pipeline-availability.check=true, hdds.scm.safemode.pipeline.creation=true, hdds.scm.safemode.threshold.pct=0.99, hdds.scm.unknown-container.action=WARN, hdds.scm.wait.time.after.safemode.exit=5m, hdds.scmclient.failover.max.retry=15, hdds.scmc
lient.failover.retry.interval=2s, hdds.scmclient.max.retry.timeout=30s, hdds.scmclient.rpc.timeout=15m, hdds.security.client.datanode.container.protocol.acl=*, hdds.security.client.scm.block.protocol.acl=*, hdds.security.client.scm.certificate.protocol.acl=*, hdds.security.client.scm.container.protocol.acl=*, hdds.security.ssl.keystore.reload.interval=60s, hdds.security.ssl.truststore.reload.interval=60s, hdds.tracing.enabled=false, hdds.x509.default.duration=P365D, hdds.x509.dir.name=certs, hdds.x509.file.name=certificate.crt, hdds.x509.max.duration=P1865D, hdds.x509.renew.grace.duration=P28D, hdds.x509.signature.algorithm=SHA256withRSA, ozone.UnsafeByteOperations.enabled=true, ozone.acl.authorizer.class=org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer, ozone.acl.enabled=true, ozone.administrators=testuser/scm@EXAMPLE.COM,testuser/s3g@EXAMPLE.COM,recon/recon@EXAMPLE.COM, ozone.block.deleting.container.limit.per.interval=10, ozone.block.deleting.limit.per.task=1000, ozone.
block.deleting.service.interval=1m, ozone.block.deleting.service.timeout=300000ms, ozone.block.deleting.service.workers=10, ozone.chunk.read.buffer.default.size=64KB, ozone.client.bucket.replication.config.refresh.time.ms=30000, ozone.client.bytes.per.checksum=1MB, ozone.client.checksum.combine.mode=COMPOSITE_CRC, ozone.client.checksum.type=CRC32, ozone.client.connection.timeout=5000ms, ozone.client.datastream.buffer.flush.size=16MB, ozone.client.datastream.min.packet.size=1MB, ozone.client.datastream.pipeline.mode=true, ozone.client.datastream.window.size=64MB, ozone.client.ec.grpc.retries.enabled=true, ozone.client.ec.grpc.retries.max=3, ozone.client.ec.reconstruct.stripe.read.pool.limit=30, ozone.client.ec.stripe.queue.size=2, ozone.client.exclude.nodes.expiry.time=600000, ozone.client.fs.default.bucket.layout=FILE_SYSTEM_OPTIMIZED, ozone.client.key.latest.version.location=true, ozone.client.key.provider.cache.expiry=10d, ozone.client.list.cache=1000, ozone.client.list.trash.keys
.max=1000, ozone.client.max.ec.stripe.write.retries=10, ozone.client.max.retries=5, ozone.client.read.timeout=30s, ozone.client.retry.interval=0, ozone.client.socket.timeout=5000ms, ozone.client.stream.buffer.flush.delay=true, ozone.client.stream.buffer.flush.size=16MB, ozone.client.stream.buffer.increment=0B, ozone.client.stream.buffer.max.size=32MB, ozone.client.stream.buffer.size=4MB, ozone.client.verify.checksum=true, ozone.container.cache.lock.stripes=1024, ozone.container.cache.size=1024, ozone.default.bucket.layout=LEGACY, ozone.directory.deleting.service.interval=1m, ozone.freon.http-address=0.0.0.0:9884, ozone.freon.http-bind-host=0.0.0.0, ozone.freon.http.auth.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, ozone.freon.http.auth.kerberos.principal=HTTP/_HOST@REALM, ozone.freon.http.auth.type=simple, ozone.freon.http.enabled=true, ozone.freon.https-address=0.0.0.0:9885, ozone.freon.https-bind-host=0.0.0.0, ozone.fs.datastream.enabled=false, ozone.fs.iterate.batch-size=10
0, ozone.fs.listing.page.size=1024, ozone.fs.listing.page.size.max=5000, ozone.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer, ozone.http.policy=HTTP_ONLY, ozone.https.client.keystore.resource=ssl-client.xml, ozone.https.client.need-auth=false, ozone.https.server.keystore.resource=ssl-server.xml, ozone.key.deleting.limit.per.task=20000, ozone.key.preallocation.max.blocks=64, ozone.manager.db.checkpoint.transfer.bandwidthPerSec=0, ozone.metadata.dirs=/data/metadata, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.metastore.rocksdb.statistics=OFF, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.network.jvm.address.cache.enabled=true, ozone.network.topology.aware.read=false, ozone.om.address=om, ozone.om.admin.protocol.max.retries=20, ozone.om.admin.protocol.wait.between.retries=1000, ozone.om.container.location.cache.size=100000, ozone.om.container.location.cache.ttl=360m, ozone.om.enable.filesystem.paths=false, ozone.om.enabl
e.ofs.shared.tmp.dir=false, ozone.om.fs.snapshot.max.limit=1000, ozone.om.grpc.maximum.response.length=134217728, ozone.om.handler.count.key=100, ozone.om.http-address=om:9874, ozone.om.http-bind-host=0.0.0.0, ozone.om.http.auth.kerberos.keytab=/etc/security/keytabs/om.keytab, ozone.om.http.auth.kerberos.principal=HTTP/om@EXAMPLE.COM, ozone.om.http.auth.type=kerberos, ozone.om.http.enabled=true, ozone.om.https-address=0.0.0.0:9875, ozone.om.https-bind-host=0.0.0.0, ozone.om.kerberos.keytab.file=/etc/security/keytabs/om.keytab, ozone.om.kerberos.principal=om/om@EXAMPLE.COM, ozone.om.key.path.lock.enabled=false, ozone.om.keyname.character.check.enabled=false, ozone.om.leader.election.minimum.timeout.duration=5s, ozone.om.multitenancy.enabled=true, ozone.om.multitenancy.ranger.sync.interval=30s, ozone.om.multitenancy.ranger.sync.timeout=10s, ozone.om.open.key.cleanup.limit.per.task=1000, ozone.om.open.key.cleanup.service.interval=24h, ozone.om.open.key.cleanup.service.timeout=300s, ozo
ne.om.open.key.expire.threshold=7d, ozone.om.ratis.enable=true, ozone.om.ratis.log.appender.queue.byte-limit=32MB, ozone.om.ratis.log.appender.queue.num-elements=1024, ozone.om.ratis.log.purge.gap=1000000, ozone.om.ratis.minimum.timeout=5s, ozone.om.ratis.port=9872, ozone.om.ratis.rpc.type=GRPC, ozone.om.ratis.segment.preallocated.size=4MB, ozone.om.ratis.segment.size=4MB, ozone.om.ratis.server.failure.timeout.duration=120s, ozone.om.ratis.server.leaderelection.pre-vote=false, ozone.om.ratis.server.request.timeout=3s, ozone.om.ratis.server.retry.cache.timeout=600000ms, ozone.om.save.metrics.interval=5m, ozone.om.security.admin.protocol.acl=*, ozone.om.security.client.protocol.acl=*, ozone.om.snapshot.cache.max.size=10, ozone.om.snapshot.compaction.dag.max.time.allowed=30d, ozone.om.snapshot.compaction.dag.prune.daemon.run.interval=3600s, ozone.om.snapshot.provider.connection.timeout=5000s, ozone.om.snapshot.provider.request.timeout=5000ms, ozone.om.snapshot.provider.socket.timeout=5
000s, ozone.om.unflushed.transaction.max.count=10000, ozone.om.user.max.volume=1024, ozone.om.volume.listall.allowed=true, ozone.path.deleting.limit.per.task=10000, ozone.recon.administrators=testuser2/scm@EXAMPLE.COM, ozone.recon.db.dir=/data/metadata/recon, ozone.recon.http-address=0.0.0.0:9888, ozone.recon.http-bind-host=0.0.0.0, ozone.recon.http.auth.kerberos.keytab=/etc/security/keytabs/recon.keytab, ozone.recon.http.auth.kerberos.principal=HTTP/recon@EXAMPLE.COM, ozone.recon.http.auth.type=kerberos, ozone.recon.http.enabled=true, ozone.recon.https-address=0.0.0.0:9889, ozone.recon.https-bind-host=0.0.0.0, ozone.recon.kerberos.keytab.file=/etc/security/keytabs/recon.keytab, ozone.recon.kerberos.principal=recon/recon@EXAMPLE.COM, ozone.recon.nssummary.flush.db.max.threshold=150000, ozone.recon.om.connection.request.timeout=5000, ozone.recon.om.connection.timeout=5s, ozone.recon.om.snapshot.task.flush.param=false, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.recon.om.sna
pshot.task.interval.delay=1m, ozone.recon.om.socket.timeout=5s, ozone.recon.scm.connection.request.timeout=5s, ozone.recon.scm.connection.timeout=5s, ozone.recon.scm.container.threshold=100, ozone.recon.scm.snapshot.enabled=true, ozone.recon.scm.snapshot.task.initial.delay=1m, ozone.recon.scm.snapshot.task.interval.delay=24h, ozone.recon.security.client.datanode.container.protocol.acl=*, ozone.recon.task.thread.count=1, ozone.replication.allowed-configs=^((STANDALONE|RATIS)/(ONE|THREE))|(EC/(3-2|6-3|10-4))$, ozone.rest.client.http.connection.max=100, ozone.rest.client.http.connection.per-route.max=20, ozone.s3g.client.buffer.size=4KB, ozone.s3g.http-address=0.0.0.0:9878, ozone.s3g.http-bind-host=0.0.0.0, ozone.s3g.http.auth.kerberos.keytab=/etc/security/keytabs/s3g.keytab, ozone.s3g.http.auth.kerberos.principal=HTTP/s3g@EXAMPLE.COM, ozone.s3g.http.auth.type=kerberos, ozone.s3g.http.enabled=true, ozone.s3g.kerberos.keytab.file=/etc/security/keytabs/s3g.keytab, ozone.s3g.kerberos.prin
cipal=s3g/s3g@EXAMPLE.COM, ozone.s3g.volume.name=s3v, ozone.scm.block.client.address=scm, ozone.scm.block.client.bind.host=0.0.0.0, ozone.scm.block.client.port=9863, ozone.scm.block.deletion.max.retry=4096, ozone.scm.block.size=256MB, ozone.scm.ca.list.retry.interval=10s, ozone.scm.chunk.size=4MB, ozone.scm.client.address=scm, ozone.scm.client.bind.host=0.0.0.0, ozone.scm.client.port=9860, ozone.scm.container.layout=FILE_PER_BLOCK2023-02-27T20:45:24.777288085Z , ozone.scm.container.placement.ec.impl=
2023-02-27 12:45:24 org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter
2023-02-27 12:45:24 , ozone.scm.container.placement.impl=
2023-02-27 12:45:24 org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom
2023-02-27 12:45:24 , ozone.scm.container.size=1GB, ozone.scm.datanode.admin.monitor.interval=30s, ozone.scm.datanode.disallow.same.peers=false, ozone.scm.datanode.id.dir=/data, ozone.scm.datanode.pipeline.limit=1, ozone.scm.datanode.port=9861, ozone.scm.datanode.ratis.volume.free-space.min=1GB, ozone.scm.dead.node.interval=45s, ozone.scm.ec.pipeline.minimum=5, ozone.scm.event.ContainerReport.thread.pool.size=10, ozone.scm.expired.container.replica.op.scrub.interval=5m, ozone.scm.grpc.port=9895, ozone.scm.ha.grpc.deadline.interval=30m, ozone.scm.ha.ratis.leader.election.timeout=5s, ozone.scm.ha.ratis.leader.ready.check.interval=2s, ozone.scm.ha.ratis.leader.ready.wait.timeout=60s, ozone.scm.ha.ratis.log.appender.queue.byte-limit=32MB, ozone.scm.ha.ratis.log.appender.queue.num-elements=1024, ozone.scm.ha.ratis.log.purge.enabled=false, ozone.scm.ha.ratis.log.purge.gap=1000000, ozone.scm.ha.ratis.request.timeout=30s, ozone.scm.ha.ratis.rpc.type=GRPC, ozone.scm.ha.ratis.segment.pr
eallocated.size=4MB, ozone.scm.ha.ratis.segment.size=4MB, ozone.scm.ha.ratis.server.failure.timeout.duration=120s, ozone.scm.ha.ratis.server.leaderelection.pre-vote=false, ozone.scm.ha.ratis.server.retry.cache.timeout=60s, ozone.scm.ha.ratis.snapshot.threshold=1000, ozone.scm.handler.count.key=100, ozone.scm.heartbeat.log.warn.interval.count=10, ozone.scm.heartbeat.rpc-retry-count=15, ozone.scm.heartbeat.rpc-retry-interval=1s, ozone.scm.heartbeat.rpc-timeout=5s, ozone.scm.heartbeat.thread.interval=3s, ozone.scm.http-address=scm:9876, ozone.scm.http-bind-host=0.0.0.0, ozone.scm.http.enabled=true, ozone.scm.https-address=0.0.0.0:9877, ozone.scm.https-bind-host=0.0.0.0, ozone.scm.info.wait.duration=10m, ozone.scm.keyvalue.container.deletion-choosing.policy=
2023-02-27 12:45:24 org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
2023-02-27 12:45:24 , ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
2023-02-27 12:45:24 org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
2023-02-27 12:45:24 , ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, scm.container.client.idle.threshold=10
s, scm.container.client.max.size=256}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] errose28 merged pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 merged PR #4271:
URL: https://github.com/apache/ozone/pull/4271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1109119842
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java:
##########
@@ -105,12 +105,12 @@ public static String toStartupShutdownString(String prefix, String... msg) {
}
public static void startupShutdownMessage(VersionInfo versionInfo,
- Class<?> clazz, String[] args, Logger log) {
+ Class<?> clazz, String[] args, Logger log, String conf) {
Review Comment:
Passing the configuration object as it is and I added a `processForLogging` utility function to redact sensitive properties and sort them before logging it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] neils-dev commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "neils-dev (via GitHub)" <gi...@apache.org>.
neils-dev commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1428466336
thanks @SaketaChalamchala - Looks to be some CI failures in the integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #4271:
URL: https://github.com/apache/ozone/pull/4271#discussion_r1105634220
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManagerStarter.java:
##########
@@ -156,7 +157,8 @@ private void commonInit() {
String[] originalArgs = getCmd().getParseResult().originalArgs()
.toArray(new String[0]);
StringUtils.startupShutdownMessage(HddsVersionInfo.HDDS_VERSION_INFO,
- StorageContainerManager.class, originalArgs, LOG);
+ StorageContainerManager.class, originalArgs, LOG,
+ conf.getAllPropertiesByTag(ConfigTag.SCM.name()).toString());
Review Comment:
There are certain property which is not properly tagged, not tagged or tagged like CONTAINER for DN, belonging to HDFS base, or belonging to RATIS, those will not be logged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1428364191
cc @sumitagrawl @dombizita @duongkame can you'll take a look as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1433876843
Similar list from other services
```
SCM :
/************************************************************
STARTUP_MSG: Starting StorageContainerManager
...
STARTUP_MSG: java = 11.0.17
STARTUP_MSG: conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, dfs.balancer.service.retries.on.exception=5, dfs.batched.ls.limit=100, dfs.block.access.key.update.interval=600, dfs.block.access.token.enable=false, dfs.block.access.token.lifetime=600, ..., dfs.datanode.balance.bandwidthPerSec=100m, dfs.datanode.balance.max.concurrent.moves=100, dfs.datanode.block-pinning.enabled=false, dfs.datanode.block.id.layout.upgrade.threads=6, dfs.datanode.bp-ready.timeout=20, dfs.datanode.cache.rev
ocation.polling.ms=500, dfs.datanode.cache.revocation.timeout.ms=900000, dfs.datanode.cached-dfsused.check.interval.ms=600000, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, dfs.datanode.data.dir.perm=700, dfs.datanode.data.transfer.bandwidthPerSec=0, dfs.datanode.data.write.bandwidthPerSec=0, dfs.datanode.directoryscan.interval=21600, dfs.datanode.directoryscan.threads=1, dfs.datanode.directoryscan.throttle.limit.ms.per.sec=1000, dfs.datanode.disk.check.min.gap=15m, ... dfs.use.dfs.network.topology=true, dfs.user.home.dir.prefix=/user, dfs.web.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, dfs.web.authentication.kerberos.principal=HTTP/ozone@EXAMPLE.COM, dfs.webhdfs.acl.provider.permission.pattern=^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$, dfs.webhdfs.netty.high.watermark=65535, dfs.webhdfs.netty.low.watermark=32768, dfs.webhdfs.oauth2.enable
d=false, dfs.webhdfs.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*, dfs.webhdfs.rest-csrf.custom-header=X-XSRF-HEADER, dfs.webhdfs.rest-csrf.enabled=false, dfs.webhdfs.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE, dfs.webhdfs.socket.connect-timeout=60s, dfs.webhdfs.socket.read-timeout=60s, dfs.webhdfs.ugi.expire.after.access=600000, dfs.webhdfs.use.ipc.callq=true, dfs.webhdfs.user.provider.user.pattern=^[A-Za-z_][A-Za-z0-9._-]*[$]?$, dfs.xframe.enabled=true, dfs.xframe.value=SAMEORIGIN, file.blocksize=67108864, file.bytes-per-checksum=512, file.client-write-packet-size=65536, file.replication=1, file.stream-buffer-size=4096, fs.AbstractFileSystem.abfs.impl=org.apache.hadoop.fs.azurebfs.Abfs, fs.AbstractFileSystem.abfss.impl=org.apache.hadoop.fs.azurebfs.Abfss, fs.AbstractFileSystem.adl.impl=org.apache.hadoop.fs.adl.Adl, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, fs.AbstractFileSystem.ftp.impl=org.apache.hadoop.fs.ftp.FtpFs, fs.AbstractFileSyst
em.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS, fs.AbstractFileSystem.har.impl=org.apache.hadoop.fs.HarFs, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, fs.AbstractFileSystem.s3a.impl=org.apache.hadoop.fs.s3a.S3A, fs.AbstractFileSystem.swebhdfs.impl=org.apache.hadoop.fs.SWebHdfs, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, fs.AbstractFileSystem.wasb.impl=org.apache.hadoop.fs.azure.Wasb, fs.AbstractFileSystem.wasbs.impl=org.apache.hadoop.fs.azure.Wasbs, fs.AbstractFileSystem.webhdfs.impl=org.apache.hadoop.fs.WebHdfs, fs.abfs.impl=org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem, fs.abfss.impl=org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem, fs.adl.impl=org.apache.hadoop.fs.adl.AdlFileSystem, fs.adl.oauth2.access.token.provider.type=<redacted>, ...
, ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
, ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}l=300s, ozone.recon.task.safemode.wait.threshold=300s, ozone.recon.task.thread.count=1, ozone.service.shutdown.timeout=60s, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10}
```
```
DN:
/************************************************************
STARTUP_MSG: Starting HddsDatanodeService
...
STARTUP_MSG: java = 11.0.17
STARTUP_MSG: conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, ..., fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.
client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, ..., ozone.scm.heartbeat.log.warn.interval.count=10, ozone.scm.heartbeat.rpc-retry-count=15, ozone.scm.heartbeat.rpc-retry-interval=1s, ozone.scm.heartbeat.rpc-timeout=5s, ozone.scm.heartbeat.thread.interval=3s, ozone.scm.http-address=scm:9876, ozone.scm.http-bind-host=0.0.0.0, ozone.scm.http.enabled=true, ozone.scm.https-address=0.0.0.0:9877, ozone.scm.https-bind-host=0.0.0.0, ozone.scm.info.wait.duration=10m, ozone.scm.keyvalue.container.deletion-choosing.policy=
org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
, ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
, ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}
```
```
S3G:
/************************************************************
...
STARTUP_MSG: java = 11.0.17
STARTUP_MSG: conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, bind.address=0.0.0.0, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, dfs.balancer.service.retries.on.exception=5, dfs.batched.ls.limit=100, dfs.block.access.key.update.interval=600, dfs.block.access.token.enable=false, dfs.block.access.token.lifetime=600, ..., fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas
.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, fs.s3a.assumed.role.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, fs.s3a.assumed.role.session.duration=30m, fs.s3a.attempts.maximum=20, fs.s3a.aws.credentials.provider=
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
, fs.s3a.block.size=32M, fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a, fs.s3a.change.detection.mode=server, fs.s3a.change.detection.source=etag, fs.s3a.change.detection.version.required=true, fs.s3a.committer.abort.pending.uploads=true, fs.s3a.committer.magic.enabled=true, fs.s3a.committer.name=file, fs.s3a.committer.staging.conflict-mode=append, fs.s3a.committer.staging.tmp.path=tmp/staging, fs.s3a.committer.staging.unique-filenames=true, fs.s3a.committer.threads=8, fs.s3a.connection.establish.timeout=5000, fs.s3a.connection.maximum=96, fs.s3a.connection.request.timeout=0, ..., ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=tr
ue, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}
```
```
Ozone CSI:
sh-4.2$ ozone csi -conf=/etc/hadoop/x --verbose
...
/************************************************************
STARTUP_MSG: Starting CsiServer
...
STARTUP_MSG: java = 11.0.17
STARTUP_MSG: conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, ..., fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.
client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, fs.s3a.assumed.role.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, fs.s3a.assumed.role.session.duration=30m, fs.s3a.attempts.maximum=20, fs.s3a.aws.credentials.provider=
org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
, fs.s3a.block.size=32M, fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a, fs.s3a.change.detection.mode=server, fs.s3a.change.detection.source=etag, fs.s3a.change.detection.version.required=true, ..., ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1447706645
One minor question:
> ```
> ## Full list of configs printed in all role logs :
> STARTUP_MSG: conf = {... ozone.scm.container.placement.ec.impl=
> 2023-02-27 12:45:24 org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter
> 2023-02-27 12:45:24 , ozone.scm.container.placement.impl=
> 2023-02-27 12:45:24 org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom
> 2023-02-27 12:45:24 , ... ozone.scm.keyvalue.container.deletion-choosing.policy=
> 2023-02-27 12:45:24 org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
> 2023-02-27 12:45:24 , ... ozone.scm.pipeline.leader-choose.policy=
> 2023-02-27 12:45:24 org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
> 2023-02-27 12:45:24 , ...}
> ```
These newlines are the result of the whitespace embedded in some config properties' default values, e.g.:
https://github.com/apache/ozone/blob/e90e2dd8ea1770bafea759015c955fc6e2281b9f/hadoop-hdds/common/src/main/resources/ozone-default.xml#L842-L845
It would be nice to ensure `conf = {...}` is printed in a single log line. Should we remove such whitespace from the specific properties in `ozone-default.xml`, or trim values programmatically in `HddsUtils#processForLogging`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] SaketaChalamchala commented on pull request #4271: HDDS-7869. Log configuration on component startup.
Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on PR #4271:
URL: https://github.com/apache/ozone/pull/4271#issuecomment-1448585271
> It would be nice to ensure `conf = {...}` is printed in a single log line. Should we remove such whitespace from the specific properties in `ozone-default.xml`, or trim values programmatically in `HddsUtils#processForLogging`?
It might be better to do trim the strings in `HddsUtils#processForLogging`. As Ethan mentioned before, we wouldn't have to worry about any new configs with whitespaces in the added in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org