You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jerryshao <gi...@git.apache.org> on 2017/08/14 07:23:39 UTC
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/18935
[SPARK-9104][CORE] Expose Netty memory metrics in Spark
## What changes were proposed in this pull request?
This PR exposes Netty memory usage for Spark's `TransportClientFactory` and `TransportServer`, including the details of each direct arena and heap arena metrics, as well as aggregated metrics. The purpose of adding the Netty metrics is to better know the memory usage of Netty in Spark shuffle, rpc and others network communications, and guide us to better configure the memory size of executors.
This PR doesn't expose these metrics to any sink, to leverage this feature, still requires to connect to either MetricsSystem or collect them back to Driver to display.
## How was this patch tested?
Add Unit test to verify it, also manually verified in real cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-9104
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18935.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18935
----
commit 05c1f4de4f00639d5f1acf1b9c061e4894d8286d
Author: jerryshao <ss...@hortonworks.com>
Date: 2017-08-14T07:16:04Z
Expose Netty memory metrics in Spark
Change-Id: I006c464674180961b12a5a86b208f113b193ff0d
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/18935
Retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80835/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81420/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80835/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80829/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@jiangxb1987 @zsxwing would you please help to review this PR, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@squito , here is the list of full metrics I dumped from running application. By default, if we don't enable verbose metric, only 2 metrics ("usedHeapMemory" and "usedDirectMemory") will be tracked. If we enable verbose metrics, then the details of each arena will be tracked, the number of detailed metrics is 16, so the total verbose metric number will be "16 * arena number * 2", here arena number is based on the number of cores, and "2" represents "direct arena" and "heap arena".
I agree this might be too verbose for a user to track netty memory usage, so by default I will not enable verbose metric. Unless if developers wants to know the detailed behaviors of network memory usage, then they could enable this configuration, this might be useful for tuning and performance analysis. What do you think?
```
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveBytes
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numActiveTinyAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numAllocations
value = 7
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numDeallocations
value = 7
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numHugeDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numNormalAllocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numNormalDeallocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numSmallAllocations
value = 3
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numSmallDeallocations
value = 3
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numTinyAllocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.directArena0.numTinyDeallocations
value = 2
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveBytes
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numActiveTinyAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numAllocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numDeallocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numHugeAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numHugeDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numNormalAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numNormalDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numSmallAllocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numSmallDeallocations
value = 0
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numTinyAllocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.heapArena0.numTinyDeallocations
value = 4
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.usedDirectMemory
value = 16777216
application_1503394031686_0002.1.NettyBlockTransfer.shuffle-client.usedHeapMemory
value = 16777216
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80641/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80618/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #81420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81420/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r134051537
--- Diff: common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java ---
@@ -106,6 +105,12 @@ public TransportClientFactory(
conf.getModuleName() + "-client");
this.pooledAllocator = NettyUtils.createPooledByteBufAllocator(
conf.preferDirectBufs(), false /* allowCache */, conf.clientThreads());
+ this.metrics = new NettyMemoryMetrics(
--- End diff --
it looks to me like this doesn't get exposed at all for the user to query -- eg. I dont' see it getting pulled into the MetricSystem. is that happening somehow? or this exposed another way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18935
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80641/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` public class TransportClientFactory implements Closeable `
* `public class NettyMemoryMetrics implements MetricSet `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18935
The change itself looks good, but I'm afraid it's a bit too verbose to add all of those metrics. cc @zsxwing .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80618/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
Hi @squito , thanks a lot for your review. This PR doesn't expose Netty metrics to MetricsSystem or collect to driver side (what did before), it just provides a way for `TransportClientFactory` and `TransportServer` to expose metrics to other components. So we can either pull this into MetricsSystem, or report to driver in the following PRs. That's why you don't find any way for user to query in this PR. This will be done it following ones.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #81420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81420/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80662/testReport)** for PR 18935 at commit [`03d0d46`](https://github.com/apache/spark/commit/03d0d46a0cd57501255d9b04b0d909ce37fef57a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class TransportClientFactory implements Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80822/testReport)** for PR 18935 at commit [`9c2b3f0`](https://github.com/apache/spark/commit/9c2b3f0a0156525dfdc4c7da1c9b7d1d41fd05f3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r133013594
--- Diff: common/network-common/src/test/java/org/apache/spark/network/util/NettyMemoryMetricsSuite.java ---
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import java.io.IOException;
+import java.util.Map;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Metric;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.MetricSet;
+import org.apache.spark.network.TestUtils;
+import org.apache.spark.network.client.TransportClient;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import org.apache.spark.network.TransportContext;
+import org.apache.spark.network.client.TransportClientFactory;
+import org.apache.spark.network.server.NoOpRpcHandler;
+import org.apache.spark.network.server.RpcHandler;
+import org.apache.spark.network.server.TransportServer;
+
+public class NettyMemoryMetricsSuite {
+
+ private TransportConf conf;
+ private TransportContext context;
+ private TransportServer server;
+ private TransportClientFactory clientFactory;
+
+ @Before
+ public void setUp() {
+ conf = new TransportConf("shuffle", MapConfigProvider.EMPTY);
+ RpcHandler rpcHandler = new NoOpRpcHandler();
+ context = new TransportContext(conf, rpcHandler);
+ server = context.createServer();
+ clientFactory = context.createClientFactory();
+ }
+
+ @After
+ public void tearDown() {
+ JavaUtils.closeQuietly(clientFactory);
+ JavaUtils.closeQuietly(server);
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testNettyMemoryMetrics() throws IOException, InterruptedException {
+ MetricSet serverMetrics = server.getAllMetrics();
+ Assert.assertNotNull(serverMetrics);
+ Assert.assertNotNull(serverMetrics.getMetrics());
+ Assert.assertNotEquals(serverMetrics.getMetrics().size(), 0);
+
+ Map<String, Metric> serverMetricMap = serverMetrics.getMetrics();
+ serverMetricMap.forEach((name, metric) -> {
+ Assert.assertTrue(name.startsWith("shuffle-server"));
+ });
--- End diff --
Indentation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80823/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@jsoltren @squito I take the liberty to get another shot of exposing Netty memory metrics. This PR doesn't address any display related things, only expose memory usage from Netty client factory and server. The following PRs could leverage this to display shuffle, RPC as well as other component's memory usage for Netty. This can be connected to `MetricsSystem`, also can be collected back to driver as you did before.
Please help to review, your comments are greatly appreciated, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/18935
thanks for the added info @jerryshao.
(a) ok makes sense now about the use of this -- it is not exposed now, you plan to expose it in future changes. That is fine, however I would like to at least see a *plan* for how you want to expose this before we merge this. I think past efforts have gotten tripped on how we'd expose it in the driver; perhaps the simplest thing to do is expose it to the metric system? There just isn't any point in this unless we're going to do something with it.
(b) thanks for including the example of the metrics. But one minor thing I noticed -- are you at all surprised that the direct & heap memory are exactly the same? Are things getting mixed up somewhere? Or maybe those are just the values from the initialization netty always does. I definitely think that is more metrics than most users care about, and would want verbose off by default. I am wondering if those metrics are useful at all ... but I guess they are pretty cheap, and maybe there is some scenario where a user would care about some of them, eg. `numHugeAllocations`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80823/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@squito can you please review again? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r133013174
--- Diff: common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java ---
@@ -57,7 +55,7 @@
* TransportClients will be reused whenever possible. Prior to completing the creation of a new
* TransportClient, all given {@link TransportClientBootstrap}s will be run.
*/
-public class TransportClientFactory implements Closeable {
+ public class TransportClientFactory implements Closeable {
--- End diff --
Could you remove the space?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80662/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18935
Just realized the last build is 18 days ago. Triggered a new now. Will merge after passing tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18935
Merging to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
Thanks @dongjoon-hyun for your review, will change it soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80618/testReport)** for PR 18935 at commit [`05c1f4d`](https://github.com/apache/spark/commit/05c1f4de4f00639d5f1acf1b9c061e4894d8286d).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` public class TransportClientFactory implements Closeable `
* `public class NettyMemoryMetrics implements MetricSet `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
Thanks @zsxwing . I was thinking to expose the details of memory allocation in Netty for user to monitor and tune, user could filter out unrelated metrics. Maybe you're right, it is too verbose and too detailed to expose each arena's detail, let me change the codes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r133874147
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -61,6 +62,7 @@ public TransportConf(String module, ConfigProvider conf) {
SPARK_NETWORK_IO_MAXRETRIES_KEY = getConfKey("io.maxRetries");
SPARK_NETWORK_IO_RETRYWAIT_KEY = getConfKey("io.retryWait");
SPARK_NETWORK_IO_LAZYFD_KEY = getConfKey("io.lazyFD");
+ SPARK_NETWORK_VERBOSE_METRICS = getConfKey("io.enableVerboseMetrics");
--- End diff --
Add a configuration to control whether to expose detailed metrics. Normally detailed per arena metrics will not be tracked, user could enable this through configuration, so that user has the ability to know the details of Netty memory allocation behavior, which could potentially guid to fine-grained tune.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80641/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80829/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r134052216
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/NettyMemoryMetrics.java ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.*;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Metric;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.MetricSet;
+import com.google.common.annotations.VisibleForTesting;
+import io.netty.buffer.PoolArenaMetric;
+import io.netty.buffer.PooledByteBufAllocator;
+import io.netty.buffer.PooledByteBufAllocatorMetric;
+
+/**
+ * A Netty memory metrics class to collect metrics from Netty PooledByteBufAllocator.
+ */
+public class NettyMemoryMetrics implements MetricSet {
+
+ private final PooledByteBufAllocator pooledAllocator;
+
+ private final boolean verboseMetricsEnabled;
+
+ private final Map<String, Metric> allMetrics;
+
+ private final String metricPrefix;
+
+ @VisibleForTesting
+ final static Set<String> VERBOSE_METRICS = new HashSet<>();
+ static {
+ VERBOSE_METRICS.addAll(Arrays.asList(
+ "numAllocations",
+ "numTinyAllocations",
+ "numSmallAllocations",
+ "numNormalAllocations",
+ "numHugeAllocations",
+ "numDeallocations",
+ "numTinyDeallocations",
+ "numSmallDeallocations",
+ "numNormalDeallocations",
+ "numHugeDeallocations",
+ "numActiveAllocations",
+ "numActiveTinyAllocations",
+ "numActiveSmallAllocations",
+ "numActiveNormalAllocations",
+ "numActiveHugeAllocations",
+ "numActiveBytes"));
+ }
+
+ public NettyMemoryMetrics(PooledByteBufAllocator pooledAllocator,
+ String metricPrefix,
+ TransportConf conf) {
+ this.pooledAllocator = pooledAllocator;
+ this.allMetrics = new HashMap<>();
+ this.metricPrefix = metricPrefix;
+ this.verboseMetricsEnabled = conf.verboseMetrics();
+
+ registerMetrics(this.pooledAllocator);
+ }
+
+ private void registerMetrics(PooledByteBufAllocator allocator) {
+ PooledByteBufAllocatorMetric pooledAllocatorMetric = allocator.metric();
+
+ // Register general metrics.
+ allMetrics.put(MetricRegistry.name(metricPrefix, "usedHeapMemory"),
+ (Gauge<Long>) () -> pooledAllocatorMetric.usedHeapMemory());
+ allMetrics.put(MetricRegistry.name(metricPrefix, "usedDirectMemory"),
+ (Gauge<Long>) () -> pooledAllocatorMetric.usedDirectMemory());
+
+ if (verboseMetricsEnabled) {
+ int directArenaIndex = 0;
+ for (PoolArenaMetric metric : pooledAllocatorMetric.directArenas()) {
+ registerArenaMetric(metric, "directArena" + directArenaIndex);
+ directArenaIndex++;
+ }
+
+ int heapArenaIndex = 0;
+ for (PoolArenaMetric metric : pooledAllocatorMetric.heapArenas()) {
+ registerArenaMetric(metric, "heapArena" + heapArenaIndex);
+ heapArenaIndex++;
+ }
+ }
+ }
+
+ private void registerArenaMetric(PoolArenaMetric arenaMetric, String arenaName) {
+ for (String methodName : VERBOSE_METRICS) {
+ Method m;
+ try {
+ m = PoolArenaMetric.class.getMethod(methodName);
+ } catch (Exception e) {
+ // Failed to find metric related method, ignore this metric.
+ continue;
+ }
+
+ if (!Modifier.isPublic(m.getModifiers())) {
+ // Ignore non-public methods.
+ continue;
+ }
+
+ Class<?> returnType = m.getReturnType();
+ String metricName = MetricRegistry.name(metricPrefix, arenaName, m.getName());
--- End diff --
also just saw zsxwings comment:
> Does anyone really care about the details of PoolArenaMetric? Maybe just report https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ByteBufAllocatorMetric.java ?
that may be enough, rather than aggregating the details from the arenas. again, an example of the info from these metrics would be helpful to decide if they're useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80829/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/18935
lgtm
any more thoughts @jiangxb1987 @zsxwing ?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18935
@squito , as the next following step, I would expose these metrics with MetricsSystem, I'm thinking of exposing shuffle related Netty memory usage. For RPC related memory usage, I'm not fully sure about the value of exposing them, though exposing them is a quite cheap call.
And furthermore, I think we can collect them in the driver side and display either through REST API or via web UI. But I think part of should be considered well on how to deliver to the end users.
As for the content of netty memory metrics, by default only two major memory usage will be exposed unless we enable verbose. I was thinking of picking some detailed metrics, but I found it is hard to decide which one is more important over others, so instead I chose to expose all of them and let user to decide which one is important. Also like you mentioned they're quite cheap and off by default, so I think it might be good to also optionally expose detailed infos.
> are you at all surprised that the direct & heap memory are exactly the same? Are things getting mixed up somewhere? Or maybe those are just the values from the initialization netty always does.
I think it doesn't mix up anything. AFAIK it is what Netty `PooledByteBufAllocator` does at initialization, this 16MB is the default chunk size. I guess at initialization, Netty will create one chunk per arena.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18935
LGTM since enableVerboseMetrics is off by default. Merging to master. Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80822/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80823/testReport)** for PR 18935 at commit [`2586802`](https://github.com/apache/spark/commit/25868023439ae1a885b73b52368d1d915cdbed2d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80822/testReport)** for PR 18935 at commit [`9c2b3f0`](https://github.com/apache/spark/commit/9c2b3f0a0156525dfdc4c7da1c9b7d1d41fd05f3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80835/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r133873922
--- Diff: pom.xml ---
@@ -571,7 +571,7 @@
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
- <version>4.0.43.Final</version>
+ <version>4.0.47.Final</version>
--- End diff --
Upgrade Netty version 4.0.47 which has the public API for `ByteBufAllocatorMetric`, what do you think @zsxwing ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18935
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/18935
Could you bump the netty version to use its new APIs rather than reflection?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18935
**[Test build #80662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80662/testReport)** for PR 18935 at commit [`03d0d46`](https://github.com/apache/spark/commit/03d0d46a0cd57501255d9b04b0d909ce37fef57a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/18935
I'll review this tomorrow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18935: [SPARK-9104][CORE] Expose Netty memory metrics in...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/18935#discussion_r134050644
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/NettyMemoryMetrics.java ---
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.util;
+
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.util.*;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Metric;
+import com.codahale.metrics.MetricRegistry;
+import com.codahale.metrics.MetricSet;
+import com.google.common.annotations.VisibleForTesting;
+import io.netty.buffer.PoolArenaMetric;
+import io.netty.buffer.PooledByteBufAllocator;
+import io.netty.buffer.PooledByteBufAllocatorMetric;
+
+/**
+ * A Netty memory metrics class to collect metrics from Netty PooledByteBufAllocator.
+ */
+public class NettyMemoryMetrics implements MetricSet {
+
+ private final PooledByteBufAllocator pooledAllocator;
+
+ private final boolean verboseMetricsEnabled;
+
+ private final Map<String, Metric> allMetrics;
+
+ private final String metricPrefix;
+
+ @VisibleForTesting
+ final static Set<String> VERBOSE_METRICS = new HashSet<>();
+ static {
+ VERBOSE_METRICS.addAll(Arrays.asList(
+ "numAllocations",
+ "numTinyAllocations",
+ "numSmallAllocations",
+ "numNormalAllocations",
+ "numHugeAllocations",
+ "numDeallocations",
+ "numTinyDeallocations",
+ "numSmallDeallocations",
+ "numNormalDeallocations",
+ "numHugeDeallocations",
+ "numActiveAllocations",
+ "numActiveTinyAllocations",
+ "numActiveSmallAllocations",
+ "numActiveNormalAllocations",
+ "numActiveHugeAllocations",
+ "numActiveBytes"));
+ }
+
+ public NettyMemoryMetrics(PooledByteBufAllocator pooledAllocator,
+ String metricPrefix,
+ TransportConf conf) {
+ this.pooledAllocator = pooledAllocator;
+ this.allMetrics = new HashMap<>();
+ this.metricPrefix = metricPrefix;
+ this.verboseMetricsEnabled = conf.verboseMetrics();
+
+ registerMetrics(this.pooledAllocator);
+ }
+
+ private void registerMetrics(PooledByteBufAllocator allocator) {
+ PooledByteBufAllocatorMetric pooledAllocatorMetric = allocator.metric();
+
+ // Register general metrics.
+ allMetrics.put(MetricRegistry.name(metricPrefix, "usedHeapMemory"),
+ (Gauge<Long>) () -> pooledAllocatorMetric.usedHeapMemory());
+ allMetrics.put(MetricRegistry.name(metricPrefix, "usedDirectMemory"),
+ (Gauge<Long>) () -> pooledAllocatorMetric.usedDirectMemory());
+
+ if (verboseMetricsEnabled) {
+ int directArenaIndex = 0;
+ for (PoolArenaMetric metric : pooledAllocatorMetric.directArenas()) {
+ registerArenaMetric(metric, "directArena" + directArenaIndex);
+ directArenaIndex++;
+ }
+
+ int heapArenaIndex = 0;
+ for (PoolArenaMetric metric : pooledAllocatorMetric.heapArenas()) {
+ registerArenaMetric(metric, "heapArena" + heapArenaIndex);
+ heapArenaIndex++;
+ }
+ }
+ }
+
+ private void registerArenaMetric(PoolArenaMetric arenaMetric, String arenaName) {
+ for (String methodName : VERBOSE_METRICS) {
+ Method m;
+ try {
+ m = PoolArenaMetric.class.getMethod(methodName);
+ } catch (Exception e) {
+ // Failed to find metric related method, ignore this metric.
+ continue;
+ }
+
+ if (!Modifier.isPublic(m.getModifiers())) {
+ // Ignore non-public methods.
+ continue;
+ }
+
+ Class<?> returnType = m.getReturnType();
+ String metricName = MetricRegistry.name(metricPrefix, arenaName, m.getName());
--- End diff --
These metrics looks interesting, but do you think having them separated out by arena is helpful? Seems like we want a view of these metrics which sums them over all arenas?
(Perhaps this is some ignornance of how arenas work in netty ... would help to seem some example metrics you get from it)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18935
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org