You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/05 16:23:39 UTC
[GitHub] [spark] xkrogen commented on a change in pull request #34079: [SPARK-36834][SHUFFLE] Add support for namespacing log lines emitted by external shuffle service
xkrogen commented on a change in pull request #34079:
URL: https://github.com/apache/spark/pull/34079#discussion_r722403290
##########
File path: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -199,11 +205,22 @@ protected void serviceInit(Configuration externalConf) throws Exception {
_conf = new Configuration(externalConf);
URL confOverlayUrl = Thread.currentThread().getContextClassLoader()
.getResource(SHUFFLE_SERVICE_CONF_OVERLAY_RESOURCE_NAME);
+ String className = YarnShuffleService.class.getName();
if (confOverlayUrl != null) {
logger.info("Initializing Spark YARN shuffle service with configuration overlay from {}",
confOverlayUrl);
_conf.addResource(confOverlayUrl);
}
+
+ String logsNamespace = _conf.get(SPARK_SHUFFLE_SERVICE_LOGS_NAMESPACE_KEY, "");
+ if (!logsNamespace.isEmpty()) {
+ logger = LoggerFactory.getLogger(className + "." + logsNamespace);
+ }
+ if (logsNamespace.contains(" ")) {
+ logger.warn("Logging namespace contains a space which might " +
+ "not be parseable by some frameworks");
Review comment:
I think we can make the message a bit more actionable by including the config name (`SPARK_SHUFFLE_SERVICE_LOGS_NAMESPACE_KEY`) and explaining the suggested format ("... It is suggested to configure this to a simple string that looks like a Java package or class name to conform to standard logger name format")
##########
File path: common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
##########
@@ -199,11 +205,22 @@ protected void serviceInit(Configuration externalConf) throws Exception {
_conf = new Configuration(externalConf);
URL confOverlayUrl = Thread.currentThread().getContextClassLoader()
.getResource(SHUFFLE_SERVICE_CONF_OVERLAY_RESOURCE_NAME);
+ String className = YarnShuffleService.class.getName();
if (confOverlayUrl != null) {
logger.info("Initializing Spark YARN shuffle service with configuration overlay from {}",
confOverlayUrl);
_conf.addResource(confOverlayUrl);
}
+
+ String logsNamespace = _conf.get(SPARK_SHUFFLE_SERVICE_LOGS_NAMESPACE_KEY, "");
+ if (!logsNamespace.isEmpty()) {
+ logger = LoggerFactory.getLogger(className + "." + logsNamespace);
+ }
+ if (logsNamespace.contains(" ")) {
+ logger.warn("Logging namespace contains a space which might " +
+ "not be parseable by some frameworks");
Review comment:
Actually while thinking about this further, maybe it doesn't make sense to log a warning. Users may not be leveraging any framework which expects logger names to look like class names, and then they will get an unnecessary warning. Perhaps we can just rely on the note in the config documentation which says to be careful about using spaces. @mridulm what do you think?
##########
File path: docs/running-on-yarn.md
##########
@@ -793,6 +793,15 @@ The following extra configuration options are available when the shuffle service
NodeManager.
</td>
</tr>
+<tr>
+ <td><code>spark.yarn.shuffle.service.logs.namespace</code></td>
+ <td><code>(not set)</code></td>
+ <td>
+ A namespace which will be appended to the class name when forming the logger name to use for
+ emitting logs from the YARN shuffle service. This would throw a warning if the namespace value
+ provided contains a space, for example, "namespace test".
Review comment:
Similar to my comment for the warning message, can we do more to explain why we encourage these naming conventions? Maybe:
```
A namespace which will be appended to the class name when forming the logger name to use for
emitting logs from the YARN shuffle service, like
`org.apache.spark.network.yarn.YarnShuffleService.logsNamespaceValue`. Since some logging frameworks
may expect the logger name to look like a class name, it's generally recommended to provide a value which
would be a valid Java package or class name and not include spaces.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org