You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/10 13:31:28 UTC
[GitHub] [spark] attilapiros opened a new pull request #33962: [SPARK-36719][WIP] Supporting Netty Logging at the network layer
attilapiros opened a new pull request #33962:
URL: https://github.com/apache/spark/pull/33962
### What changes were proposed in this pull request?
Supporting Netty level logging at the network layer.
To configure Netty level logging a `LogHandler` must be added to the channel pipeline.
In this PR I have introduced a new class `NettyLogger` which is able to construct a log handler depending on the log level:
- in case of "log4j.logger.org.apache.spark.network.util.NettyLogger=DEBUG": a custom log handler is created which does not dumps the message contents. This way the log is a bit more compact. Moreover when the network level encryption is switched on this logger can be used.
- in case of "log4j.logger.org.apache.spark.network.util.NettyLogger=TRACE": Netty's own log handler is used which dumps the message contents.
- otherwise (when the logger is not TRACE or DEBUG) the pipeline does not contain a log handler (so this the default behaviour).
### Why are the changes needed?
This level of logging proved to be sufficient during debugging some external shuffle related problem.
And compared with the tcpdump this log lines can be more easily correlated with the Spark internal calls.
Moreover the log layout can be configured to contain the thread names that way in case of a timeout a busy thread could be found.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually.
#### DEBUG level
```
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$ tail -1 ./conf/log4j.properties
log4j.logger.org.apache.spark.network.util.NettyLogger=DEBUG
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$ ./bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master local\[8\] ./examples/target/original-spark-examples_2.12-3.3.0-SNAPSHOT.jar README.md 2> >(grep NettyLogger) 1> /dev/null
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf] REGISTERED
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf] CONNECT: /172.30.64.219:61014
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] ACTIVE
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 - R:/172.30.64.219:61015] REGISTERED
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 - R:/172.30.64.219:61015] ACTIVE
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] WRITE 66B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] FLUSH
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 - R:/172.30.64.219:61015] READ 66B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 - R:/172.30.64.219:61015] WRITE: MessageWithHeader [headerLength: 74, bodyLength: 1552705]
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 - R:/172.30.64.219:61015] FLUSH
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 74B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 - R:/172.30.64.219:61015] READ COMPLETE
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ COMPLETE
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 2048B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 32768B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ COMPLETE
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 65536B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ 10561B
21/09/10 15:24:35 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ COMPLETE
21/09/10 15:24:40 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 ! R:/172.30.64.219:61015] INACTIVE
21/09/10 15:24:40 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 - R:/172.30.64.219:61014] READ COMPLETE
21/09/10 15:24:40 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 ! R:/172.30.64.219:61014] INACTIVE
21/09/10 15:24:40 DEBUG NettyLogger: [id: 0xb9d94fcf, L:/172.30.64.219:61015 ! R:/172.30.64.219:61014] UNREGISTERED
21/09/10 15:24:40 DEBUG NettyLogger: [id: 0x28101520, L:/172.30.64.219:61014 ! R:/172.30.64.219:61015] UNREGISTERED
```
#### TRACE level
```
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$ tail -1 ./conf/log4j.properties
log4j.logger.org.apache.spark.network.util.NettyLogger=TRACE
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$ ./bin/spark-submit --class org.apache.spark.examples.JavaWordCount --master local\[8\] ./examples/target/original-spark-examples_2.12-3.3.0-SNAPSHOT.jar README.md 1> /dev/null 2>&1
...
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786] REGISTERED
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786] CONNECT: /172.30.64.219:61044
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786, L:/172.30.64.219:61045 - R:/172.30.64.219:61044] ACTIVE
21/09/10 15:29:14 INFO TransportClientFactory: Successfully created connection to /172.30.64.219:61044 after 37 ms (0 ms spent in bootstraps)
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] REGISTERED
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] ACTIVE
21/09/10 15:29:14 INFO Utils: Fetching spark://172.30.64.219:61044/jars/original-spark-examples_2.12-3.3.0-SNAPSHOT.jar to /private/var/folders/t_/fr_vqcyx23vftk81ftz1k5hw0000gn/T/spark-91e059f5-1e29-4727-8602-f81206bbe48b/userFiles-50b48490-8950-4c46-b3d3-61a2c85412a3/fetchFileTemp8803030587223485061.tmp
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786, L:/172.30.64.219:61045 - R:/172.30.64.219:61044] WRITE: 66B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 00 00 00 00 00 00 00 42 06 00 00 00 35 2f 6a 61 |.......B....5/ja|
|00000010| 72 73 2f 6f 72 69 67 69 6e 61 6c 2d 73 70 61 72 |rs/original-spar|
|00000020| 6b 2d 65 78 61 6d 70 6c 65 73 5f 32 2e 31 32 2d |k-examples_2.12-|
|00000030| 33 2e 33 2e 30 2d 53 4e 41 50 53 48 4f 54 2e 6a |3.3.0-SNAPSHOT.j|
|00000040| 61 72 |ar |
+--------+-------------------------------------------------+----------------+
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786, L:/172.30.64.219:61045 - R:/172.30.64.219:61044] FLUSH
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] READ: 66B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 00 00 00 00 00 00 00 42 06 00 00 00 35 2f 6a 61 |.......B....5/ja|
|00000010| 72 73 2f 6f 72 69 67 69 6e 61 6c 2d 73 70 61 72 |rs/original-spar|
|00000020| 6b 2d 65 78 61 6d 70 6c 65 73 5f 32 2e 31 32 2d |k-examples_2.12-|
|00000030| 33 2e 33 2e 30 2d 53 4e 41 50 53 48 4f 54 2e 6a |3.3.0-SNAPSHOT.j|
|00000040| 61 72 |ar |
+--------+-------------------------------------------------+----------------+
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] WRITE: MessageWithHeader [headerLength: 74, bodyLength: 1552705]
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] FLUSH
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786, L:/172.30.64.219:61045 - R:/172.30.64.219:61044] READ: 74B
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33962: [SPARK-36719][CORE][WIP] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-916951231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47655/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917454691
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917368456
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47666/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706449849
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyLogger.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufHolder;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.handler.logging.LoggingHandler;
+import io.netty.handler.logging.LogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NettyLogger {
+ private static final Logger logger = LoggerFactory.getLogger(NettyLogger.class);
+
+ /** A Netty LoggingHandler which does not dump the message contents. */
+ private static class NoContentLoggingHandler extends LoggingHandler {
+
+ NoContentLoggingHandler(Class<?> clazz, LogLevel level) {
+ super(clazz, level);
+ }
+
+ protected String format(ChannelHandlerContext ctx, String eventName, Object arg) {
+ if (arg instanceof ByteBuf) {
+ return format(ctx, eventName) + " " + ((ByteBuf) arg).readableBytes() + "B";
Review comment:
super nit: Add a space between value and "B" (here and below) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917476365
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143169/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917362832
> So far I haven't seen any specific logger which is documented. Probably because loggers are more for the developers who are doing the debugging and not for the users. But if you can point me to a document sure I will extend it.
Hm, I only see a logging related section "Configuring Logging" in "Spark Configuration" https://spark.apache.org/docs/latest/configuration.html#configuring-logging. Okay to skip it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917361130
**[Test build #143162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143162/testReport)** for PR 33962 at commit [`1104d6e`](https://github.com/apache/spark/commit/1104d6ec6b4bb676ec24188bb5442c4cd5cfaec2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE][WIP] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-916935353
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47655/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706333677
##########
File path: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
##########
@@ -187,7 +190,11 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- ChannelPipeline pipeline = channel.pipeline()
+ ChannelPipeline pipeline = channel.pipeline();
+ if (nettyLogger.getLoggingHandler() != null) {
+ pipeline.addLast("loggingHandler", nettyLogger.getLoggingHandler());
Review comment:
Yes as in this case we are logging what goes into the wire and not what goes between two channel handers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917187076
+CC @otterc
You have worked on similar in past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917452836
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706331130
##########
File path: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
##########
@@ -187,7 +190,11 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- ChannelPipeline pipeline = channel.pipeline()
+ ChannelPipeline pipeline = channel.pipeline();
+ if (nettyLogger.getLoggingHandler() != null) {
+ pipeline.addLast("loggingHandler", nettyLogger.getLoggingHandler());
Review comment:
Just a question, this is intentionally added at the first, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917367301
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47666/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917471415
**[Test build #143169 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143169/testReport)** for PR 33962 at commit [`1104d6e`](https://github.com/apache/spark/commit/1104d6ec6b4bb676ec24188bb5442c4cd5cfaec2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706329748
##########
File path: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
##########
@@ -187,7 +190,11 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- ChannelPipeline pipeline = channel.pipeline()
+ ChannelPipeline pipeline = channel.pipeline();
+ if (nettyLogger.getLoggingHandler() != null) {
Review comment:
When the log level is neither DEBUG nor TRACE:
https://github.com/apache/spark/blob/c5a05d8425f6fa14b34f81469730b43403bdddd7/common/network-common/src/main/java/org/apache/spark/network/util/NettyLogger.java#L58
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-916910621
**[Test build #143151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143151/testReport)** for PR 33962 at commit [`c5a05d8`](https://github.com/apache/spark/commit/c5a05d8425f6fa14b34f81469730b43403bdddd7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33962: [SPARK-36719][CORE][WIP] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-916951231
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47655/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917444648
**[Test build #143169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143169/testReport)** for PR 33962 at commit [`1104d6e`](https://github.com/apache/spark/commit/1104d6ec6b4bb676ec24188bb5442c4cd5cfaec2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917042179
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706577301
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyLogger.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufHolder;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.handler.logging.LoggingHandler;
+import io.netty.handler.logging.LogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NettyLogger {
+ private static final Logger logger = LoggerFactory.getLogger(NettyLogger.class);
+
+ /** A Netty LoggingHandler which does not dump the message contents. */
+ private static class NoContentLoggingHandler extends LoggingHandler {
+
+ NoContentLoggingHandler(Class<?> clazz, LogLevel level) {
+ super(clazz, level);
+ }
+
+ protected String format(ChannelHandlerContext ctx, String eventName, Object arg) {
+ if (arg instanceof ByteBuf) {
+ return format(ctx, eventName) + " " + ((ByteBuf) arg).readableBytes() + "B";
+ } else if (arg instanceof ByteBufHolder) {
+ return format(ctx, eventName) + " " +
+ ((ByteBufHolder) arg).content().readableBytes() + "B";
+ } else {
+ return super.format(ctx, eventName, arg);
+ }
+ }
+ }
+
+ private final LoggingHandler loggingHandler;
+
+ public NettyLogger() {
+ if (logger.isTraceEnabled()) {
+ loggingHandler = new LoggingHandler(NettyLogger.class, LogLevel.TRACE);
+ } else if (logger.isDebugEnabled()) {
+ loggingHandler = new NoContentLoggingHandler(NettyLogger.class, LogLevel.DEBUG);
+ } else {
+ loggingHandler = null;
+ }
+ }
+
+ public LoggingHandler getLoggingHandler(){
Review comment:
nit: add a space before `{`, i.e. `getLoggingHandler(){` -> `getLoggingHandler() {`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917368456
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47666/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917499718
Thank you, @attilapiros , @mridulm , @viirya .
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #33962:
URL: https://github.com/apache/spark/pull/33962
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917377871
**[Test build #143162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143162/testReport)** for PR 33962 at commit [`1104d6e`](https://github.com/apache/spark/commit/1104d6ec6b4bb676ec24188bb5442c4cd5cfaec2).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917454691
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917361130
**[Test build #143162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143162/testReport)** for PR 33962 at commit [`1104d6e`](https://github.com/apache/spark/commit/1104d6ec6b4bb676ec24188bb5442c4cd5cfaec2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917444648
**[Test build #143169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143169/testReport)** for PR 33962 at commit [`1104d6e`](https://github.com/apache/spark/commit/1104d6ec6b4bb676ec24188bb5442c4cd5cfaec2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][WIP] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-916910621
**[Test build #143151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143151/testReport)** for PR 33962 at commit [`c5a05d8`](https://github.com/apache/spark/commit/c5a05d8425f6fa14b34f81469730b43403bdddd7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706328737
##########
File path: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
##########
@@ -187,7 +190,11 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- ChannelPipeline pipeline = channel.pipeline()
+ ChannelPipeline pipeline = channel.pipeline();
+ if (nettyLogger.getLoggingHandler() != null) {
Review comment:
When does this become `null`? According to the PR description, when the logger is not TRACE or DEBUG?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917361945
I have done a quick search and no markdown contains any relevant mention of DEBUG/TRACE:
```
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$ find . -name "*.md" -exec grep DEBUG {} \; -print
enable extra logging of Kerberos operations in Hadoop by setting the `HADOOP_JAAS_DEBUG`
export HADOOP_JAAS_DEBUG=true
spark.yarn.appMasterEnv.HADOOP_JAAS_DEBUG true
Finally, if the log level for `org.apache.spark.deploy.yarn.Client` is set to `DEBUG`, the log
./docs/running-on-yarn.md
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$ find . -name "*.md" -exec grep TRACE {} \; -print
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/spark ‹SPARK-36719*›
╰─$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706328737
##########
File path: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
##########
@@ -187,7 +190,11 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- ChannelPipeline pipeline = channel.pipeline()
+ ChannelPipeline pipeline = channel.pipeline();
+ if (nettyLogger.getLoggingHandler() != null) {
Review comment:
When does this become `null`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917022742
**[Test build #143151 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143151/testReport)** for PR 33962 at commit [`c5a05d8`](https://github.com/apache/spark/commit/c5a05d8425f6fa14b34f81469730b43403bdddd7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NettyLogger `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706570973
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyLogger.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufHolder;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.handler.logging.LoggingHandler;
+import io.netty.handler.logging.LogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NettyLogger {
+ private static final Logger logger = LoggerFactory.getLogger(NettyLogger.class);
+
+ /** A Netty LoggingHandler which does not dump the message contents. */
+ private static class NoContentLoggingHandler extends LoggingHandler {
+
+ NoContentLoggingHandler(Class<?> clazz, LogLevel level) {
+ super(clazz, level);
+ }
+
+ protected String format(ChannelHandlerContext ctx, String eventName, Object arg) {
+ if (arg instanceof ByteBuf) {
+ return format(ctx, eventName) + " " + ((ByteBuf) arg).readableBytes() + "B";
Review comment:
That is consistent with Netty's own logging.
You can see it in the TRACE output in the PR description.
But I copy it for you (the last line is the proof):
```
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 00 00 00 00 00 00 00 42 06 00 00 00 35 2f 6a 61 |.......B....5/ja|
|00000010| 72 73 2f 6f 72 69 67 69 6e 61 6c 2d 73 70 61 72 |rs/original-spar|
|00000020| 6b 2d 65 78 61 6d 70 6c 65 73 5f 32 2e 31 32 2d |k-examples_2.12-|
|00000030| 33 2e 33 2e 30 2d 53 4e 41 50 53 48 4f 54 2e 6a |3.3.0-SNAPSHOT.j|
|00000040| 61 72 |ar |
+--------+-------------------------------------------------+----------------+
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786, L:/172.30.64.219:61045 - R:/172.30.64.219:61044] FLUSH
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] READ: 66B
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706570973
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/NettyLogger.java
##########
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufHolder;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.handler.logging.LoggingHandler;
+import io.netty.handler.logging.LogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NettyLogger {
+ private static final Logger logger = LoggerFactory.getLogger(NettyLogger.class);
+
+ /** A Netty LoggingHandler which does not dump the message contents. */
+ private static class NoContentLoggingHandler extends LoggingHandler {
+
+ NoContentLoggingHandler(Class<?> clazz, LogLevel level) {
+ super(clazz, level);
+ }
+
+ protected String format(ChannelHandlerContext ctx, String eventName, Object arg) {
+ if (arg instanceof ByteBuf) {
+ return format(ctx, eventName) + " " + ((ByteBuf) arg).readableBytes() + "B";
Review comment:
This (no space) is consistent with Netty's own logging.
You can see it in the TRACE output in the PR description.
But I copy it for you (the last line is the proof):
```
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 00 00 00 00 00 00 00 42 06 00 00 00 35 2f 6a 61 |.......B....5/ja|
|00000010| 72 73 2f 6f 72 69 67 69 6e 61 6c 2d 73 70 61 72 |rs/original-spar|
|00000020| 6b 2d 65 78 61 6d 70 6c 65 73 5f 32 2e 31 32 2d |k-examples_2.12-|
|00000030| 33 2e 33 2e 30 2d 53 4e 41 50 53 48 4f 54 2e 6a |3.3.0-SNAPSHOT.j|
|00000040| 61 72 |ar |
+--------+-------------------------------------------------+----------------+
21/09/10 15:29:14 TRACE NettyLogger: [id: 0xf1d25786, L:/172.30.64.219:61045 - R:/172.30.64.219:61044] FLUSH
21/09/10 15:29:14 TRACE NettyLogger: [id: 0x362fc693, L:/172.30.64.219:61044 - R:/172.30.64.219:61045] READ: 66B
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917454679
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47673/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917042179
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917440448
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917378167
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917476365
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143169/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917378167
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143162/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #33962:
URL: https://github.com/apache/spark/pull/33962#discussion_r706577441
##########
File path: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
##########
@@ -44,9 +44,11 @@
import org.apache.spark.network.server.TransportServerBootstrap;
import org.apache.spark.network.util.IOMode;
import org.apache.spark.network.util.NettyUtils;
+import org.apache.spark.network.util.NettyLogger;
import org.apache.spark.network.util.TransportConf;
import org.apache.spark.network.util.TransportFrameDecoder;
+
Review comment:
remove this extra newline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917361424
@viirya Thanks for the review!
> Is there any document we should add this `log4j.logger.org.apache.spark.network.util.NettyLogger`?
So far I haven't seen any specific logger which is documented. Probably because loggers are more for the developers who are doing the debugging and not for the users. But if you can point me to a document sure I will extend it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33962: [SPARK-36719][CORE] Supporting Netty Logging at the network layer
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33962:
URL: https://github.com/apache/spark/pull/33962#issuecomment-917368453
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47666/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org