You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "bhattmanish98 (via GitHub)" <gi...@apache.org> on 2025/12/16 11:54:59 UTC
[PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
bhattmanish98 opened a new pull request, #8137:
URL: https://github.com/apache/hadoop/pull/8137
This PR introduces a centralized Aggregated Metrics Manager and defines the conditions under which aggregated metrics are emitted from individual file systems.
Key Changes
1. **Criteria for Emitting Aggregated Metrics**
Aggregated metrics are emitted based on the following conditions:
- _Time-based interval_ - Each file system periodically emits its collected metrics at a fixed interval. After emission, metric collection is reset.
- _Threshold-based emission_ - A scheduler runs at regular intervals to check whether the total number of operations has exceeded a configured threshold. This prevents the aggregated metrics string from growing too large to be safely sent as an HTTP request header. If the threshold is reached, the collected metrics are emitted immediately, and metric collection is reset.
- _Idle-period emission_ - If a file system remains idle for a configured duration, any accumulated metrics are emitted, and metric collection is reset.
- _File system close_ - When a file system is closed, all remaining collected metrics are emitted to ensure no data is lost.
2. **Centralized Metrics Management**
All file systems now push their aggregated metrics to a shared Aggregated Metrics Manager. This manager evaluates the configured emission criteria and determines whether metrics should be emitted immediately or deferred until a later time.
This will also rate limit the number of metrics calls per second.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650764851
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAggregateMetricsManager.java:
##########
@@ -0,0 +1,710 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+
+public class TestAggregateMetricsManager extends AbstractAbfsIntegrationTest {
+
+ /** Number of nanoseconds in one millisecond. */
+ private static final long NANOS_PER_MILLISECOND = 1_000_000L;
+
+ // The manager under test
+ private final AggregateMetricsManager manager;
+
+ /**
+ * Constructor for TestAggregateMetricsManager.
+ *
+ * @throws Exception if an error occurs during setup
+ */
+ public TestAggregateMetricsManager() throws Exception {
+ super();
+ manager = AggregateMetricsManager.get(10, 3);
+ }
+
+ /**
+ * Creates a fully instrumented TracingContext useful for metric dispatch tests.
+ */
+ private String getMetricsData() {
+ return UUID.randomUUID() + COLON + UUID.randomUUID() + COLON + "#BO:";
+ }
+
+ /**
+ * Wraps an AbfsClient instance in a spy and counts metric call invocations.
+ */
+ private AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,
+ AtomicInteger counter)
+ throws IOException {
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ counter.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+
+ return client;
+ }
+
+ /**
+ * Verifies that multiple recordMetric calls result in exactly one aggregated
+ * dispatch within a 1-second dispatch window.
+ */
+ @Test
+ public void testRecordMetric() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 1 aggregated metrics send")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Verifies that aggregated metrics for the same account are sent
+ * once per dispatch window, meaning 2 windows → 2 sends.
+ */
+ @Test
+ public void testRecordMetricTwoWindows() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+
+ manager.deregisterClient("acc1", client);
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 1 aggregated sends")
+ .isEqualTo(1);
+
+ manager.registerClient("acc1", client);
+ // Second window
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 2 aggregated sends")
+ .isEqualTo(2);
+ }
+
+ /**
+ * Ensures that recordMetric handles invalid input without exceptions.
+ */
+ @Test
+ public void testRecordMetricWithNulls() throws Exception {
+ manager.recordMetric(null, null);
+ manager.recordMetric("", null);
+ manager.recordMetric("acc", null);
+ manager.recordMetric("acc", null);
+ }
+
+ /**
+ * Ensures that metrics for separate accounts still respect global
+ * rate limiting but send independently within the same window.
+ */
+ @Test
+ public void testMultipleAccounts() throws Exception {
+
+ AtomicInteger calls1 = new AtomicInteger();
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger();
+ AbfsClient client2 = spyClient(Mockito.spy(
+ (AzureBlobFileSystem) AzureBlobFileSystem.newInstance(
+ getRawConfiguration())), calls2);
+
+ manager.registerClient("acc1", client1);
+ manager.registerClient("acc2", client2);
+ manager.recordMetric("acc1", getMetricsData());
+ manager.recordMetric("acc2", getMetricsData());
+ manager.deregisterClient("acc1", client1);
+ manager.deregisterClient("acc2", client2);
+
+ Assertions.assertThat(calls1.get())
+ .describedAs("Account 1 dispatched once")
+ .isEqualTo(1);
+
+ Assertions.assertThat(calls2.get())
+ .describedAs("Account 2 dispatched once")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Tests concurrent registration, metric recording, and deregistration
+ * of multiple clients for the same account.
+ */
+ @Test
+ public void testMultipleClientsRegistryInParallel() throws Exception {
+ AtomicInteger calls1 = new AtomicInteger(0);
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger(0);
+ AbfsClient client2 = spyClient(Mockito.spy(this.getFileSystem()), calls2);
+
+ CountDownLatch latch = new CountDownLatch(5);
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ for (int i = 0; i < 10; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ //wait for all threads to finish
+ latch.await();
+
+ Assertions.assertThat((calls1.get() == 1) ^ (calls2.get() == 1))
+ .describedAs("Exactly one client should send metrics")
+ .isTrue();
+ }
+
+ /**
+ * Tests deregistering a nonexistent client.
+ */
+ @Test
+ public void testDeregisterNonexistentClient() throws IOException {
+ AbfsClient client = Mockito.spy(
+ this.getFileSystem().getAbfsStore().getClient());
+ // Should not throw
+ boolean isRemoved = manager.deregisterClient("nonexistentAccount", client);
+ Assertions.assertThat(isRemoved)
+ .describedAs("Deregistering nonexistent client should return false")
+ .isFalse();
+ }
+
+ /**
+ * Tests that when the aggregated metric data exceeds the buffer size,
+ * multiple dispatches occur as expected.
+ */
+ @Test
+ public void testMultipleMetricCallsInCaseDataIsMoreThanBufferSize()
+ throws Exception {
+ final int metricsDataSize1
+ = 927; // size of aggregated data for first 3 calls
+ final int metricsDataSize2 = 115; // size of aggregated data for last call
+ final int numberOfMetrics = 25; // total metrics to send
+ AtomicInteger calls = new AtomicInteger(0);
+ AzureBlobFileSystem azureBlobFileSystem = Mockito.spy(this.getFileSystem());
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ String data = (String) inv.getArguments()[0];
+ if (calls.get() < 3) { // first three calls, data size will be 927 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 927 chars")
+ .isEqualTo(metricsDataSize1);
+ } else { // last call, data size will be 115 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 115 chars")
+ .isEqualTo(metricsDataSize2);
+ }
+ calls.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < numberOfMetrics; i++) {
+ manager.recordMetric("acc1", getMetricsData()
+ + "$OT=163$RT=6.024%$TRNR=2543$TR=2706"); // each data is 113 chars
+ }
+ manager.deregisterClient("acc1", client);
+
+ // 113 + 2 ([,]) = 115 chars per metric, 115 * 25 = 2875 chars total + 24 (:) = 2899 chars
+ // 1st -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 2nd -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 3rd -> 115 * 9 = 920 chars + 7 (:) = 927 chars
+ // 4th -> remaining
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 3 aggregated metrics send")
+ .isEqualTo(4);
+ }
+
+ /**
+ * Verifies that when multiple clients send metrics concurrently,
+ * the global rate limiter enforces spacing between dispatches.
+ */
+ @Test
+ public void testRateLimitMetricCalls()
+ throws IOException, InterruptedException {
+ final int permitsPerSecond = 3;
+ final long minIntervalMs = 1_000 / permitsPerSecond; // 333ms
+ final double toleranceMs = 15; // allow 15ms jitter
+ final int numClients = 10;
+
+ // Store timestamps for each client
+ final List<AtomicLong> times = new ArrayList<>();
+ AbfsClient[] abfsClients = new AbfsClient[numClients];
+
+ for (int i = 0; i < numClients; i++) {
+ AtomicLong time = new AtomicLong();
+ times.add(time);
+
+ AbfsClient client = createSpiedClient(time);
+ abfsClients[i] = client;
+ manager.registerClient("acc" + i, client);
+ }
+
+ // Record metrics for all clients
+ for (int i = 0; i < numClients; i++) {
+ manager.recordMetric("acc" + i, getMetricsData());
+ }
+
+ // Deregister all clients concurrently
+ CountDownLatch latch = new CountDownLatch(numClients);
+ for (int i = 0; i < numClients; i++) {
+ final int idx = i;
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc" + idx,
+ abfsClients[idx]); // pass time for demonstration if needed
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+ }
+ latch.await();
+
+ // Check that interval between any two timestamps is ≥ minIntervalMs - tolerance
+ for (int i = 0; i < times.size(); i++) {
+ for (int j = i + 1; j < times.size(); j++) {
+ double diffMs = Math.abs(times.get(i).get() - times.get(j).get())
+ / (double) NANOS_PER_MILLISECOND;
+ Assertions.assertThat(diffMs)
+ .describedAs(
+ "Expected at least %d ms (tolerance %.3f) between metric sends",
+ minIntervalMs, toleranceMs)
+ .isGreaterThanOrEqualTo(minIntervalMs - toleranceMs);
+ }
+ }
+ }
+
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit0()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 1 (kept exactly as you asked)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " AzureBlobFileSystem fs = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AbfsClient client = spyClient(Mockito.spy(fs), calls1);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client);\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " System.exit(0);\n"
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,\n"
+ + " AtomicInteger counter) throws IOException {\n"
+ + "\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + "\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, true, 0);
+ }
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit after multiple
+ * clients and deregistrations.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit1()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 2 (final, multi-FS version)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AtomicInteger calls2 = new AtomicInteger();\n"
+ + " AtomicInteger calls3 = new AtomicInteger();\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " AzureBlobFileSystem fs1 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs2 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs3 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + "\n"
+ + " AbfsClient client1 = spyClient(Mockito.spy(fs1), calls1);\n"
+ + " AbfsClient client2 = spyClient(Mockito.spy(fs2), calls2);\n"
+ + " AbfsClient client3 = spyClient(Mockito.spy(fs3), calls3);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client1);\n"
+ + " mgr.registerClient(\"acc1\", client2);\n"
+ + " mgr.registerClient(\"acc1\", client3);\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m3\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m4\");\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m5\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m6\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " mgr.deregisterClient(\"acc1\", client3);\n"
+ + " mgr.deregisterClient(\"acc1\", client2);\n"
+ + " mgr.deregisterClient(\"acc1\", client1);\n"
+ + " System.out.println(\"BEFORE_EXIT1\");\n"
+ + " System.exit(1);\n"
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,\n"
+ + " AtomicInteger counter) throws IOException {\n"
+ + "\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + "\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, true, 1);
+ }
+
+ /**
+ * Tests that the shutdown hook does not flush metrics on JVM crash.
+ */
+ @Test
+ void testAggregatedMetricsManagerWithJVMCrash() throws Exception {
+ final int crashExitCode = 134;
+ // -------------------------------
+ // Program 3 (JVM Crash)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import java.lang.reflect.*;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + "\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " // Track how many times metrics flush\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AtomicInteger calls2 = new AtomicInteger();\n"
+ + " AtomicInteger calls3 = new AtomicInteger();\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " // Initialize 3 separate file system instances\n"
+ + " AzureBlobFileSystem fs1 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs2 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs3 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + "\n"
+ + " // Create 3 spy clients\n"
+ + " AbfsClient client1 = spyClient(Mockito.spy(fs1), calls1);\n"
+ + " AbfsClient client2 = spyClient(Mockito.spy(fs2), calls2);\n"
+ + " AbfsClient client3 = spyClient(Mockito.spy(fs3), calls3);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client1);\n"
+ + " mgr.registerClient(\"acc2\", client2);\n"
+ + " mgr.registerClient(\"acc3\", client3);\n"
+ + "\n"
+ + " // Produce metrics on all clients\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + " mgr.recordMetric(\"acc2\", \"m3\");\n"
+ + " mgr.recordMetric(\"acc2\", \"m4\");\n"
+ + " mgr.recordMetric(\"acc3\", \"m5\");\n"
+ + " mgr.recordMetric(\"acc3\", \"m6\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " crashJvm();\n"
+ + " }\n"
+ + "\n"
+ + " private static void crashJvm() throws Exception {\n"
+ + " Field f = sun.misc.Unsafe.class.getDeclaredField(\"theUnsafe\");\n"
+ + " f.setAccessible(true);\n"
+ + " sun.misc.Unsafe unsafe = (sun.misc.Unsafe) f.get(null);\n"
+ + " unsafe.putAddress(0, 0); // SIGSEGV → Immediate JVM crash\n"
+ // 128 + 6 = 134 (exitcode for SIGABRT)
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem, AtomicInteger counter) throws IOException {\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, false, crashExitCode);
+ }
+
+ /**
+ * Compiles and runs a Java program in a separate JVM, capturing its output.
+ *
+ * @param program The Java program source code as a string.
+ * @param expectMetricsFlush Whether to expect metrics flush output.
+ * @throws IOException If an I/O error occurs.
+ * @throws InterruptedException If the thread is interrupted while waiting.
+ */
+ private void runProgramAndCaptureOutput(String program,
+ boolean expectMetricsFlush, int expectedExitCode)
+ throws IOException, InterruptedException {
+ Path tempFile = Files.createTempFile("ShutdownTestProg", ".java");
Review Comment:
These temporary files are never deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650700026
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
+ // This client is the last one — drain metrics now
+ batchToSend = metricsBuffer.getAndSet(new ConcurrentLinkedQueue<>());
+ isLastClient = true;
+ }
+
+ clients.remove(client);
+ }
+ if (isLastClient) {
+ sendMetrics(client, batchToSend);
+ }
+ return true;
+ }
+
+ /**
+ * Add a metric to the buffer.
+ * @param metric the metric to add
+ */
+ void addRequest(String metric) {
+ if (metric != null) {
+ metricsBuffer.get().add(metric);
+ }
+ }
+
+ /**
+ * Drain the metrics buffer and send if there are registered clients.
+ */
+ public void drainAndSendIfReady() {
+ AbfsClient client;
+
+ synchronized (this) {
+ if (clients.isEmpty()) {
+ return;
+ }
+ client = clients.iterator().next();
+ }
+
+ ConcurrentLinkedQueue<String> batch = metricsBuffer.getAndSet(
+ new ConcurrentLinkedQueue<>());
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ sendMetrics(client, batch);
+ }
+
+ // Send metrics outside synchronized block
+ private void sendMetrics(AbfsClient client,
+ ConcurrentLinkedQueue<String> batchToSend) {
+ // Send outside synchronized block
+ if (client != null && batchToSend != null && !batchToSend.isEmpty()) {
+ for (String chunk : splitListBySize(batchToSend, MAX_HEADER_SIZE)) {
+ rateLimiter.acquire(); // Rate limiting
+ try {
+ client.getMetricCall(chunk);
+ } catch (IOException ignored) {}
+ }
+ }
+ }
+
+ // Check if there are no registered clients
+ public boolean isEmpty() {
+ return clients.isEmpty();
+ }
+
+ /**
+ * Split the list of metrics into chunks that fit within maxChars.
+ * Each metric is wrapped in square brackets and separated by colons.
+ */
+ private static List<String> splitListBySize(
+ ConcurrentLinkedQueue<String> items, long maxChars) {
+
+ if (items.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ List<String> result = new ArrayList<>();
+ StringBuilder sb = new StringBuilder();
+
+ for (String s : new ArrayList<>(items)) {
Review Comment:
No need of copy here, you can directly iterate as for (String s: items)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2648071903
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ this.rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleAtFixedRate(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
Review Comment:
Shouldn't we wait for dispatch metrics to finish before scheduler is shutdown ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647809304
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##########
@@ -338,7 +339,7 @@ void completeExecute(TracingContext tracingContext)
LOG.debug("Rest operation {} failed with failureReason: {}. Retrying with retryCount = {}, retryPolicy: {} and sleepInterval: {}",
operationType, failureReason, retryCount, retryPolicy.getAbbreviation(), retryInterval);
if (abfsBackoffMetrics != null) {
- updateBackoffTimeMetrics(retryCount, sleepDuration);
+ updateBackoffTimeMetrics(retryCount, retryInterval);
Review Comment:
Earlier sleepDuration was wrongly passed in updateBackoffTimeMetrics, actually it is retryInterval which tells us the delay between two retries.
https://github.com/apache/hadoop/blob/4a82720b30839e570d74339f9f300763c30ed020/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652199762
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
Review Comment:
nit: break is redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650701576
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
+ // This client is the last one — drain metrics now
+ batchToSend = metricsBuffer.getAndSet(new ConcurrentLinkedQueue<>());
+ isLastClient = true;
+ }
+
+ clients.remove(client);
+ }
+ if (isLastClient) {
+ sendMetrics(client, batchToSend);
+ }
+ return true;
+ }
+
+ /**
+ * Add a metric to the buffer.
+ * @param metric the metric to add
+ */
+ void addRequest(String metric) {
+ if (metric != null) {
+ metricsBuffer.get().add(metric);
+ }
+ }
+
+ /**
+ * Drain the metrics buffer and send if there are registered clients.
+ */
+ public void drainAndSendIfReady() {
+ AbfsClient client;
+
+ synchronized (this) {
+ if (clients.isEmpty()) {
+ return;
+ }
+ client = clients.iterator().next();
+ }
+
+ ConcurrentLinkedQueue<String> batch = metricsBuffer.getAndSet(
+ new ConcurrentLinkedQueue<>());
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ sendMetrics(client, batch);
+ }
+
+ // Send metrics outside synchronized block
+ private void sendMetrics(AbfsClient client,
+ ConcurrentLinkedQueue<String> batchToSend) {
+ // Send outside synchronized block
+ if (client != null && batchToSend != null && !batchToSend.isEmpty()) {
+ for (String chunk : splitListBySize(batchToSend, MAX_HEADER_SIZE)) {
+ rateLimiter.acquire(); // Rate limiting
+ try {
+ client.getMetricCall(chunk);
+ } catch (IOException ignored) {}
+ }
+ }
+ }
+
+ // Check if there are no registered clients
+ public boolean isEmpty() {
Review Comment:
should be synchronized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3722045910
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 20s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 23s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/7/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 25s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 46s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| -1 :x: | javadoc | 0m 32s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/7/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 23s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 106m 21s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 81a41bf58fe6 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3f8af3d2d6f9c5281ca305319c4f2e445e5ac54d |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/7/testReport/ |
| Max. process+thread count | 640 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/7/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2667151256
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
Review Comment:
Even then we may need two constructors as it is getting used in some test cases. But as you suggest I can simplify this if else statement in initializeReadFooterMetrics method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2665153908
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/SimpleRateLimiter.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+
+public final class SimpleRateLimiter {
+
+ // Interval between permits in nanoseconds.
+ private final long intervalNanos;
+
+ // Next allowed time to acquire a permit in nanoseconds.
+ private long nextAllowedTime;
Review Comment:
Not needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2672019659
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsMetricsManager.java:
##########
@@ -0,0 +1,396 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.net.URL;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidUriException;
+import org.apache.hadoop.fs.azurebfs.utils.MetricFormat;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.apache.hadoop.fs.azurebfs.utils.UriUtils;
+
+import static org.apache.commons.lang3.StringUtils.isNotEmpty;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemUriSchemes.HTTPS_SCHEME;
+import static org.apache.hadoop.fs.azurebfs.enums.AbfsBackoffMetricsEnum.TOTAL_NUMBER_OF_REQUESTS;
+import static org.apache.hadoop.fs.azurebfs.services.AbfsClient.LOG;
+
+/**
+ * AbfsMetricsManager is responsible for managing metrics collection
+ * and emission for an AbfsClient instance.
+ */
+public class AbfsMetricsManager implements Closeable {
+
+ // Timer thread name for AbfsMetricsManager
+ public static final String ABFS_CLIENT_TIMER_THREAD_NAME
+ = "abfs-timer-client";
+
+ // Timer for scheduling metric emission tasks based on idle time
+ private Timer timer;
+
+ // URL for sending metrics
+ private URL metricUrl;
+
+ // Shared key credentials for metric account
+ private SharedKeyCredentials metricSharedkeyCredentials = null;
+
+ // Currently running TimerTask
+ private TimerTask runningTimerTask;
+
+ // Metric analysis periods
+ private final int metricAnalysisPeriod;
+
+ // Metric idle period
+ private final int metricIdlePeriod;
+
+ // Flag to indicate if a separate metric account is used
+ private boolean hasSeparateMetricAccount = false;
+
+ // Flag to indicate if metric collection is enabled
+ private final AtomicBoolean isMetricCollectionEnabled
+ = new AtomicBoolean(false);
+
+ // Metric format for metrics
+ private MetricFormat metricFormat;
+
+ // Flag to indicate if metric collection is stopped
+ private final AtomicBoolean isMetricCollectionStopped;
+
+ // AggregateMetricsManager instance
+ private final AggregateMetricsManager aggregateMetricsManager;
+
+ // Scheduler to emit aggregated metric based on time
+ private ScheduledExecutorService metricsEmitScheduler = null;
+
+ // AbfsConfiguration instance
+ private final AbfsConfiguration abfsConfiguration;
+
+ // AbfsCounters instance
+ private final AbfsCounters abfsCounters;
+
+ // File system ID
+ private final String fileSystemId;
+
+ // Storage account name
+ private final String accountName;
+
+ /**
+ * Constructor for AbfsMetricsManager.
+ *
+ * @param abfsConfiguration AbfsConfiguration object.
+ * @param abfsCounters AbfsCounters object.
+ * @param baseUrlString Base URL string of the AbfsClient.
+ * @param indexLastForwardSlash Index of last forward slash in the base URL string.
+ * @param accountName Storage account name.
+ * @param fileSystemId File system ID.
+ */
+ public AbfsMetricsManager(final AbfsConfiguration abfsConfiguration,
+ final AbfsCounters abfsCounters, final String baseUrlString,
+ final int indexLastForwardSlash, final String accountName,
+ final String fileSystemId) {
+ this.abfsConfiguration = abfsConfiguration;
+ this.abfsCounters = abfsCounters;
+ this.fileSystemId = fileSystemId;
+ this.isMetricCollectionEnabled.set(
+ abfsConfiguration.isMetricsCollectionEnabled());
+ this.isMetricCollectionStopped = new AtomicBoolean(false);
+ this.aggregateMetricsManager = AggregateMetricsManager.getInstance(
+ abfsConfiguration.getMetricsEmitIntervalInMins(),
+ abfsConfiguration.getMaxMetricsCallsPerSecond());
+ this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
+ this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
+ this.accountName = accountName;
+ if (isMetricCollectionEnabled()) {
+ try {
+ String metricAccountName = abfsConfiguration.getMetricAccount();
+ String metricAccountKey = abfsConfiguration.getMetricAccountKey();
+ this.metricFormat = abfsConfiguration.getMetricFormat();
+ if (isNotEmpty(metricAccountName) && isNotEmpty(
+ metricAccountKey)) {
+ int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ metricAccountName + " - account name is not fully qualified.");
+ }
+ try {
+ metricSharedkeyCredentials = new SharedKeyCredentials(
+ metricAccountName.substring(0, dotIndex),
+ metricAccountKey);
+ hasSeparateMetricAccount = true;
+ setMetricsUrl(metricAccountName.startsWith(HTTPS_SCHEME)
+ ? metricAccountName : HTTPS_SCHEME + COLON
+ + FORWARD_SLASH + FORWARD_SLASH + metricAccountName);
+ } catch (IllegalArgumentException e) {
+ throw new IOException(
+ "Exception while initializing metric credentials ", e);
+ }
+ } else {
+ setMetricsUrl(baseUrlString.substring(0, indexLastForwardSlash + 1));
+ }
+ // Once the metric URL is set, initialize the metrics
+ abfsCounters.initializeMetrics(metricFormat, abfsConfiguration);
+ // Metrics emitter scheduler
+ this.metricsEmitScheduler
+ = Executors.newSingleThreadScheduledExecutor();
+ // run every 1 minute to check the metrics count
+ this.metricsEmitScheduler.scheduleWithFixedDelay(
+ () -> {
+ if (abfsCounters.getAbfsBackoffMetrics()
+ .getMetricValue(TOTAL_NUMBER_OF_REQUESTS)
+ >= abfsConfiguration.getMetricsEmitThreshold()) {
+ emitCollectedMetrics();
+ }
+ },
+ abfsConfiguration.getMetricsEmitThresholdIntervalInSecs(),
+ abfsConfiguration.getMetricsEmitThresholdIntervalInSecs(),
+ TimeUnit.SECONDS);
+
+ // run every metricInterval minutes
+ this.metricsEmitScheduler.scheduleWithFixedDelay(
+ this::emitCollectedMetrics,
+ abfsConfiguration.getMetricsEmitIntervalInMins(),
+ abfsConfiguration.getMetricsEmitIntervalInMins(),
+ TimeUnit.MINUTES);
+
+ // emit metrics based on idea time
+ if (abfsConfiguration.shouldEmitMetricsOnIdleTime()) {
+ this.timer = new Timer(
+ ABFS_CLIENT_TIMER_THREAD_NAME, true);
+ timer.schedule(new TimerTaskImpl(),
+ metricIdlePeriod,
+ metricIdlePeriod);
+ }
+ } catch (Exception e) {
+ LOG.error("Metrics disabled. Failed to initialize metrics for {}",
+ baseUrlString, e);
+ this.isMetricCollectionEnabled.set(false);
+ }
+ }
+ }
+
+ /**
+ * Closes the metrics resources.
+ * This method cancels any running timer tasks, shuts down the metrics emission scheduler,
+ * and emits any collected metrics before closing.
+ */
+ @Override
+ public void close() {
+ if (runningTimerTask != null) {
+ runningTimerTask.cancel();
+ }
+ if (timer != null) {
Review Comment:
if (timer != null) {
timer.cancel();
timer.purge();
timer = null;
} https://github.com/apache/hadoop/pull/7852/ as per this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3660589025
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 6s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 6s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| -1 :x: | javadoc | 0m 45s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 39s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 22s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 23s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 25 new + 5 unchanged - 0 fixed = 30 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 22s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | shadedclient | 26m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 106m 35s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Switch statement found in org.apache.hadoop.fs.azurebfs.AbfsCountersImpl.initializeMetrics(MetricFormat) where one case falls through to the next case At AbfsCountersImpl.java:where one case falls through to the next case At AbfsCountersImpl.java:[lines 210-212] |
| | org.apache.hadoop.fs.azurebfs.services.MetricsBucket.sendMetrics(AbfsClient, ConcurrentLinkedQueue) might ignore java.io.IOException At MetricsBucket.java: At MetricsBucket.java:[line 140] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 9e2fbc887cf4 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / abf1f1e90be6961e80bc486a2fedd3c763ceba59 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/testReport/ |
| Max. process+thread count | 610 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/1/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3675226589
============================================================
HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 904, Failures: 0, Errors: 0, Skipped: 220
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
HNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 907, Failures: 0, Errors: 0, Skipped: 166
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10
============================================================
NonHNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 744, Failures: 0, Errors: 0, Skipped: 287
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11
============================================================
AppendBlob-HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 904, Failures: 0, Errors: 0, Skipped: 231
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-SharedKey-Blob
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 750, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11
============================================================
NonHNS-OAuth-DFS
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 741, Failures: 0, Errors: 0, Skipped: 289
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 747, Failures: 0, Errors: 0, Skipped: 156
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
AppendBlob-NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 742, Failures: 0, Errors: 0, Skipped: 202
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
HNS-Oauth-DFS-IngressBlob
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 778, Failures: 0, Errors: 0, Skipped: 229
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-OAuth-DFS-IngressBlob
============================================================
[WARNING] Tests run: 235, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 739, Failures: 0, Errors: 0, Skipped: 286
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650759169
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAggregateMetricsManager.java:
##########
@@ -0,0 +1,710 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+
+public class TestAggregateMetricsManager extends AbstractAbfsIntegrationTest {
+
+ /** Number of nanoseconds in one millisecond. */
+ private static final long NANOS_PER_MILLISECOND = 1_000_000L;
+
+ // The manager under test
+ private final AggregateMetricsManager manager;
+
+ /**
+ * Constructor for TestAggregateMetricsManager.
+ *
+ * @throws Exception if an error occurs during setup
+ */
+ public TestAggregateMetricsManager() throws Exception {
+ super();
+ manager = AggregateMetricsManager.get(10, 3);
+ }
+
+ /**
+ * Creates a fully instrumented TracingContext useful for metric dispatch tests.
+ */
+ private String getMetricsData() {
+ return UUID.randomUUID() + COLON + UUID.randomUUID() + COLON + "#BO:";
+ }
+
+ /**
+ * Wraps an AbfsClient instance in a spy and counts metric call invocations.
+ */
+ private AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,
+ AtomicInteger counter)
+ throws IOException {
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ counter.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+
+ return client;
+ }
+
+ /**
+ * Verifies that multiple recordMetric calls result in exactly one aggregated
+ * dispatch within a 1-second dispatch window.
+ */
+ @Test
+ public void testRecordMetric() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 1 aggregated metrics send")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Verifies that aggregated metrics for the same account are sent
+ * once per dispatch window, meaning 2 windows → 2 sends.
+ */
+ @Test
+ public void testRecordMetricTwoWindows() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+
+ manager.deregisterClient("acc1", client);
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 1 aggregated sends")
+ .isEqualTo(1);
+
+ manager.registerClient("acc1", client);
+ // Second window
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 2 aggregated sends")
+ .isEqualTo(2);
+ }
+
+ /**
+ * Ensures that recordMetric handles invalid input without exceptions.
+ */
+ @Test
+ public void testRecordMetricWithNulls() throws Exception {
+ manager.recordMetric(null, null);
+ manager.recordMetric("", null);
+ manager.recordMetric("acc", null);
+ manager.recordMetric("acc", null);
+ }
+
+ /**
+ * Ensures that metrics for separate accounts still respect global
+ * rate limiting but send independently within the same window.
+ */
+ @Test
+ public void testMultipleAccounts() throws Exception {
+
+ AtomicInteger calls1 = new AtomicInteger();
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger();
+ AbfsClient client2 = spyClient(Mockito.spy(
+ (AzureBlobFileSystem) AzureBlobFileSystem.newInstance(
+ getRawConfiguration())), calls2);
+
+ manager.registerClient("acc1", client1);
+ manager.registerClient("acc2", client2);
+ manager.recordMetric("acc1", getMetricsData());
+ manager.recordMetric("acc2", getMetricsData());
+ manager.deregisterClient("acc1", client1);
+ manager.deregisterClient("acc2", client2);
+
+ Assertions.assertThat(calls1.get())
+ .describedAs("Account 1 dispatched once")
+ .isEqualTo(1);
+
+ Assertions.assertThat(calls2.get())
+ .describedAs("Account 2 dispatched once")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Tests concurrent registration, metric recording, and deregistration
+ * of multiple clients for the same account.
+ */
+ @Test
+ public void testMultipleClientsRegistryInParallel() throws Exception {
+ AtomicInteger calls1 = new AtomicInteger(0);
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger(0);
+ AbfsClient client2 = spyClient(Mockito.spy(this.getFileSystem()), calls2);
+
+ CountDownLatch latch = new CountDownLatch(5);
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ for (int i = 0; i < 10; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ //wait for all threads to finish
+ latch.await();
+
+ Assertions.assertThat((calls1.get() == 1) ^ (calls2.get() == 1))
+ .describedAs("Exactly one client should send metrics")
+ .isTrue();
+ }
+
+ /**
+ * Tests deregistering a nonexistent client.
+ */
+ @Test
+ public void testDeregisterNonexistentClient() throws IOException {
+ AbfsClient client = Mockito.spy(
+ this.getFileSystem().getAbfsStore().getClient());
+ // Should not throw
+ boolean isRemoved = manager.deregisterClient("nonexistentAccount", client);
+ Assertions.assertThat(isRemoved)
+ .describedAs("Deregistering nonexistent client should return false")
+ .isFalse();
+ }
+
+ /**
+ * Tests that when the aggregated metric data exceeds the buffer size,
+ * multiple dispatches occur as expected.
+ */
+ @Test
+ public void testMultipleMetricCallsInCaseDataIsMoreThanBufferSize()
+ throws Exception {
+ final int metricsDataSize1
+ = 927; // size of aggregated data for first 3 calls
+ final int metricsDataSize2 = 115; // size of aggregated data for last call
+ final int numberOfMetrics = 25; // total metrics to send
+ AtomicInteger calls = new AtomicInteger(0);
+ AzureBlobFileSystem azureBlobFileSystem = Mockito.spy(this.getFileSystem());
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ String data = (String) inv.getArguments()[0];
+ if (calls.get() < 3) { // first three calls, data size will be 927 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 927 chars")
+ .isEqualTo(metricsDataSize1);
+ } else { // last call, data size will be 115 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 115 chars")
+ .isEqualTo(metricsDataSize2);
+ }
+ calls.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < numberOfMetrics; i++) {
+ manager.recordMetric("acc1", getMetricsData()
+ + "$OT=163$RT=6.024%$TRNR=2543$TR=2706"); // each data is 113 chars
+ }
+ manager.deregisterClient("acc1", client);
+
+ // 113 + 2 ([,]) = 115 chars per metric, 115 * 25 = 2875 chars total + 24 (:) = 2899 chars
+ // 1st -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 2nd -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 3rd -> 115 * 9 = 920 chars + 7 (:) = 927 chars
+ // 4th -> remaining
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 3 aggregated metrics send")
+ .isEqualTo(4);
+ }
+
+ /**
+ * Verifies that when multiple clients send metrics concurrently,
+ * the global rate limiter enforces spacing between dispatches.
+ */
+ @Test
+ public void testRateLimitMetricCalls()
+ throws IOException, InterruptedException {
+ final int permitsPerSecond = 3;
+ final long minIntervalMs = 1_000 / permitsPerSecond; // 333ms
+ final double toleranceMs = 15; // allow 15ms jitter
+ final int numClients = 10;
+
+ // Store timestamps for each client
+ final List<AtomicLong> times = new ArrayList<>();
+ AbfsClient[] abfsClients = new AbfsClient[numClients];
+
+ for (int i = 0; i < numClients; i++) {
+ AtomicLong time = new AtomicLong();
+ times.add(time);
+
+ AbfsClient client = createSpiedClient(time);
+ abfsClients[i] = client;
+ manager.registerClient("acc" + i, client);
+ }
+
+ // Record metrics for all clients
+ for (int i = 0; i < numClients; i++) {
+ manager.recordMetric("acc" + i, getMetricsData());
+ }
+
+ // Deregister all clients concurrently
+ CountDownLatch latch = new CountDownLatch(numClients);
+ for (int i = 0; i < numClients; i++) {
+ final int idx = i;
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc" + idx,
+ abfsClients[idx]); // pass time for demonstration if needed
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+ }
+ latch.await();
+
+ // Check that interval between any two timestamps is ≥ minIntervalMs - tolerance
+ for (int i = 0; i < times.size(); i++) {
+ for (int j = i + 1; j < times.size(); j++) {
+ double diffMs = Math.abs(times.get(i).get() - times.get(j).get())
+ / (double) NANOS_PER_MILLISECOND;
+ Assertions.assertThat(diffMs)
+ .describedAs(
+ "Expected at least %d ms (tolerance %.3f) between metric sends",
+ minIntervalMs, toleranceMs)
+ .isGreaterThanOrEqualTo(minIntervalMs - toleranceMs);
+ }
+ }
+ }
+
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit0()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 1 (kept exactly as you asked)
Review Comment:
remove this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647722292
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -281,42 +303,84 @@ private AbfsClient(final URL baseUrl,
new ThreadFactoryBuilder().setNameFormat("AbfsClient Lease Ops").setDaemon(true).build();
this.executorService = MoreExecutors.listeningDecorator(
HadoopExecutors.newScheduledThreadPool(this.abfsConfiguration.getNumLeaseThreads(), tf));
- this.metricFormat = abfsConfiguration.getMetricFormat();
+ this.isMetricCollectionEnabled.set(abfsConfiguration.isMetricsCollectionEnabled());
this.isMetricCollectionStopped = new AtomicBoolean(false);
this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
- if (StringUtils.isNotEmpty(metricFormat.toString())) {
- String metricAccountName = abfsConfiguration.getMetricAccount();
- String metricAccountKey = abfsConfiguration.getMetricAccountKey();
- if (StringUtils.isNotEmpty(metricAccountName) && StringUtils.isNotEmpty(metricAccountKey)) {
- isMetricCollectionEnabled = true;
- abfsCounters.initializeMetrics(metricFormat);
- int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- metricAccountName + " - account name is not fully qualified.");
+ if (isMetricCollectionEnabled()) {
+ try {
+ String metricAccountName = abfsConfiguration.getMetricAccount();
+ String metricAccountKey = abfsConfiguration.getMetricAccountKey();
+ this.metricFormat = abfsConfiguration.getMetricFormat();
+ abfsCounters.initializeMetrics(metricFormat, getAbfsConfiguration());
+ if (isNotEmpty(metricAccountName) && isNotEmpty(
+ metricAccountKey)) {
+ int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ metricAccountName + " - account name is not fully qualified.");
+ }
+ try {
+ metricSharedkeyCredentials = new SharedKeyCredentials(
+ metricAccountName.substring(0, dotIndex),
+ metricAccountKey);
+ hasSeparateMetricAccount = true;
+ setMetricsUrl(metricAccountName.startsWith(HTTPS_SCHEME)
+ ? metricAccountName : HTTPS_SCHEME + COLON
+ + FORWARD_SLASH + FORWARD_SLASH + metricAccountName);
+ } catch (IllegalArgumentException e) {
+ throw new IOException(
+ "Exception while initializing metric credentials ", e);
+ }
+ } else {
+ setMetricsUrl(baseUrlString.substring(0, indexLastForwardSlash + 1));
}
- try {
- metricSharedkeyCredentials = new SharedKeyCredentials(
- metricAccountName.substring(0, dotIndex),
- metricAccountKey);
- } catch (IllegalArgumentException e) {
- throw new IOException("Exception while initializing metric credentials ", e);
+
+ // register the client to Aggregated Metrics Manager
+ this.aggregateMetricsManager.registerClient(accountName, this);
+
+ // Metrics emitter scheduler
+ this.metricsEmitScheduler
+ = Executors.newSingleThreadScheduledExecutor();
+ // run every 1 minute to check the metrics count
+ this.metricsEmitScheduler.scheduleAtFixedRate(
Review Comment:
Yes, it is as per design, each file system will emit the metrics to manager class at regular interval if not closed. The singleton manager class will do actual API call to send those collected metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647760456
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -281,42 +303,84 @@ private AbfsClient(final URL baseUrl,
new ThreadFactoryBuilder().setNameFormat("AbfsClient Lease Ops").setDaemon(true).build();
this.executorService = MoreExecutors.listeningDecorator(
HadoopExecutors.newScheduledThreadPool(this.abfsConfiguration.getNumLeaseThreads(), tf));
- this.metricFormat = abfsConfiguration.getMetricFormat();
+ this.isMetricCollectionEnabled.set(abfsConfiguration.isMetricsCollectionEnabled());
this.isMetricCollectionStopped = new AtomicBoolean(false);
this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
- if (StringUtils.isNotEmpty(metricFormat.toString())) {
- String metricAccountName = abfsConfiguration.getMetricAccount();
- String metricAccountKey = abfsConfiguration.getMetricAccountKey();
- if (StringUtils.isNotEmpty(metricAccountName) && StringUtils.isNotEmpty(metricAccountKey)) {
- isMetricCollectionEnabled = true;
- abfsCounters.initializeMetrics(metricFormat);
- int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- metricAccountName + " - account name is not fully qualified.");
+ if (isMetricCollectionEnabled()) {
Review Comment:
Okay, let me check the feasibility of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647756122
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##########
@@ -136,6 +136,13 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_ENABLE_AUTOTHROTTLING = false;
public static final int DEFAULT_METRIC_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS = 60_000;
+ public static final boolean DEFAULT_METRICS_COLLECTION_ENABLED = true;
Review Comment:
Yes, this feature will be enabled by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2648019050
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
+ }
+ }
+
+ /**
+ * Initialize the read footer metrics.
+ * In case the metrics are already initialized,
+ * create a new instance with the existing map.
+ */
+ private void initializeReadFooterMetrics() {
+ if (abfsReadFooterMetrics == null) {
+ abfsReadFooterMetrics = new AbfsReadFooterMetrics();
+ } else {
+ //In case metrics is emitted based on total count, there could be a chance
+ // that file type for which we have calculated the type will be lost.
+ // To avoid that, creating a new instance with existing map.
+ abfsReadFooterMetrics = new AbfsReadFooterMetrics(
Review Comment:
Not able to get this.
How can the file type will be lost? It must be a part of AbfsReadFooterMetrics object only
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
Review Comment:
Nit: Empty line changes can be reverted.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
+ private static final String FOOTER_LENGTH = "20";
+
+ private static final List<FileType> FILE_TYPE_LIST =
+ Arrays.asList(FileType.values());
+
+ private Map<String, FileTypeMetrics> fileTypeMetricsMap =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Inner class to handle file type checks.
+ */
+ private static final class FileTypeMetrics {
+
+ private final AtomicBoolean collectMetrics;
+
+ private final AtomicBoolean collectMetricsForNextRead;
+
+ private final AtomicBoolean collectLenMetrics;
+
+ private final AtomicLong readCount;
+
+ private final AtomicLong offsetOfFirstRead;
+
+ private FileType fileType = null;
+
+ private String sizeReadByFirstRead;
+
+ private String offsetDiffBetweenFirstAndSecondRead;
/**
- * Inner class to handle file type checks.
+ * Constructor to initialize the file type metrics.
*/
- private static final class FileTypeMetrics {
- private final AtomicBoolean collectMetrics;
- private final AtomicBoolean collectMetricsForNextRead;
- private final AtomicBoolean collectLenMetrics;
- private final AtomicLong readCount;
- private final AtomicLong offsetOfFirstRead;
- private FileType fileType = null;
- private String sizeReadByFirstRead;
- private String offsetDiffBetweenFirstAndSecondRead;
-
- /**
- * Constructor to initialize the file type metrics.
- */
- private FileTypeMetrics() {
- collectMetrics = new AtomicBoolean(false);
- collectMetricsForNextRead = new AtomicBoolean(false);
- collectLenMetrics = new AtomicBoolean(false);
- readCount = new AtomicLong(0);
- offsetOfFirstRead = new AtomicLong(0);
- }
-
- /**
- * Updates the file type based on the metrics collected.
- */
- private void updateFileType() {
- if (fileType == null) {
- fileType = collectMetrics.get() && readCount.get() >= 2
- && haveEqualValues(sizeReadByFirstRead)
- && haveEqualValues(offsetDiffBetweenFirstAndSecondRead) ? PARQUET : NON_PARQUET;
- }
- }
-
- /**
- * Checks if the given value has equal parts.
- *
- * @param value the value to check
- * @return true if the value has equal parts, false otherwise
- */
- private boolean haveEqualValues(String value) {
- String[] parts = value.split("_");
- return parts.length == 2
- && parts[0].equals(parts[1]);
- }
-
- /**
- * Increments the read count.
- */
- private void incrementReadCount() {
- readCount.incrementAndGet();
- }
-
- /**
- * Returns the read count.
- *
- * @return the read count
- */
- private long getReadCount() {
- return readCount.get();
- }
-
- /**
- * Sets the collect metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetrics(boolean collect) {
- collectMetrics.set(collect);
- }
-
- /**
- * Returns the collect metrics flag.
- *
- * @return the collect metrics flag
- */
- private boolean getCollectMetrics() {
- return collectMetrics.get();
- }
-
- /**
- * Sets the collect metrics for the next read flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetricsForNextRead(boolean collect) {
- collectMetricsForNextRead.set(collect);
- }
-
- /**
- * Returns the collect metrics for the next read flag.
- *
- * @return the collect metrics for the next read flag
- */
- private boolean getCollectMetricsForNextRead() {
- return collectMetricsForNextRead.get();
- }
-
- /**
- * Returns the collect length metrics flag.
- *
- * @return the collect length metrics flag
- */
- private boolean getCollectLenMetrics() {
- return collectLenMetrics.get();
- }
-
- /**
- * Sets the collect length metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectLenMetrics(boolean collect) {
- collectLenMetrics.set(collect);
- }
-
- /**
- * Sets the offset of the first read.
- *
- * @param offset the value to set
- */
- private void setOffsetOfFirstRead(long offset) {
- offsetOfFirstRead.set(offset);
- }
-
- /**
- * Returns the offset of the first read.
- *
- * @return the offset of the first read
- */
- private long getOffsetOfFirstRead() {
- return offsetOfFirstRead.get();
- }
-
- /**
- * Sets the size read by the first read.
- *
- * @param size the value to set
- */
- private void setSizeReadByFirstRead(String size) {
- sizeReadByFirstRead = size;
- }
-
- /**
- * Returns the size read by the first read.
- *
- * @return the size read by the first read
- */
- private String getSizeReadByFirstRead() {
- return sizeReadByFirstRead;
- }
-
- /**
- * Sets the offset difference between the first and second read.
- *
- * @param offsetDiff the value to set
- */
- private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
- offsetDiffBetweenFirstAndSecondRead = offsetDiff;
- }
-
- /**
- * Returns the offset difference between the first and second read.
- *
- * @return the offset difference between the first and second read
- */
- private String getOffsetDiffBetweenFirstAndSecondRead() {
- return offsetDiffBetweenFirstAndSecondRead;
- }
-
- /**
- * Returns the file type.
- *
- * @return the file type
- */
- private FileType getFileType() {
- return fileType;
- }
+ private FileTypeMetrics() {
+ collectMetrics = new AtomicBoolean(false);
+ collectMetricsForNextRead = new AtomicBoolean(false);
+ collectLenMetrics = new AtomicBoolean(false);
+ readCount = new AtomicLong(0);
+ offsetOfFirstRead = new AtomicLong(0);
}
/**
- * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ * Updates the file type based on the metrics collected.
Review Comment:
Explain in javadoc what logic is used to infer file type
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingHeaderVersion.java:
##########
@@ -47,7 +47,11 @@ public enum TracingHeaderVersion {
* :position:operatedBlobCount:operationSpecificHeader:httpOperationHeader
* :aggregatedMetrics:resourceUtilizationMetrics
*/
- V2("v2", 15);
+ V2("v2", 15),
+ /**
+ * Metrics to client request id header.
+ */
+ AV0("av0", 3);
Review Comment:
Add what 3 fields will be there in comment as added in other versions javadoc
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
+ }
+ }
+
+ /**
+ * Initialize the read footer metrics.
+ * In case the metrics are already initialized,
+ * create a new instance with the existing map.
+ */
+ private void initializeReadFooterMetrics() {
+ if (abfsReadFooterMetrics == null) {
+ abfsReadFooterMetrics = new AbfsReadFooterMetrics();
+ } else {
+ //In case metrics is emitted based on total count, there could be a chance
Review Comment:
Typo, space after //
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
Review Comment:
Instead of passing AbfsConfiguration here, better to only pass isRetryMetricsEnabled.
Whole object is not needed
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsReadFooterMetrics.java:
##########
@@ -57,32 +54,14 @@
import org.apache.hadoop.fs.azurebfs.utils.TracingHeaderValidator;
import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
-import static org.assertj.core.api.Assumptions.assumeThat;
-
public class ITestAbfsReadFooterMetrics extends AbstractAbfsScaleTest {
- public ITestAbfsReadFooterMetrics() throws Exception {
- checkPrerequisites();
- }
-
- private void checkPrerequisites(){
Review Comment:
Are these moved somewhere else?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
+ }
+ }
+
+ /**
+ * Initialize the read footer metrics.
+ * In case the metrics are already initialized,
+ * create a new instance with the existing map.
Review Comment:
Why create a new instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2665177979
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
Review Comment:
It is as per best practice to keep break statement with default.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
Review Comment:
It is as per best practice to keep break statement with default case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3719368394
============================================================
HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 906, Failures: 0, Errors: 0, Skipped: 220
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
HNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 909, Failures: 0, Errors: 0, Skipped: 166
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 10
============================================================
NonHNS-SharedKey-DFS
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 746, Failures: 0, Errors: 0, Skipped: 287
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11
============================================================
AppendBlob-HNS-OAuth-DFS
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 906, Failures: 0, Errors: 0, Skipped: 231
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-SharedKey-Blob
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 754, Failures: 0, Errors: 0, Skipped: 144
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 11
============================================================
NonHNS-OAuth-DFS
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 743, Failures: 0, Errors: 0, Skipped: 289
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 751, Failures: 0, Errors: 0, Skipped: 156
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
AppendBlob-NonHNS-OAuth-Blob
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 746, Failures: 0, Errors: 0, Skipped: 202
[WARNING] Tests run: 135, Failures: 0, Errors: 0, Skipped: 4
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
============================================================
HNS-Oauth-DFS-IngressBlob
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 780, Failures: 0, Errors: 0, Skipped: 229
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 8
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 23
============================================================
NonHNS-OAuth-DFS-IngressBlob
============================================================
[WARNING] Tests run: 237, Failures: 0, Errors: 0, Skipped: 10
[WARNING] Tests run: 743, Failures: 0, Errors: 0, Skipped: 286
[WARNING] Tests run: 158, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 271, Failures: 0, Errors: 0, Skipped: 24
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2672221769
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsMetricsManager.java:
##########
@@ -0,0 +1,396 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.net.URL;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.fs.azurebfs.AbfsConfiguration;
+import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidUriException;
+import org.apache.hadoop.fs.azurebfs.utils.MetricFormat;
+import org.apache.hadoop.fs.azurebfs.utils.TracingContext;
+import org.apache.hadoop.fs.azurebfs.utils.UriUtils;
+
+import static org.apache.commons.lang3.StringUtils.isNotEmpty;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.FORWARD_SLASH;
+import static org.apache.hadoop.fs.azurebfs.constants.FileSystemUriSchemes.HTTPS_SCHEME;
+import static org.apache.hadoop.fs.azurebfs.enums.AbfsBackoffMetricsEnum.TOTAL_NUMBER_OF_REQUESTS;
+import static org.apache.hadoop.fs.azurebfs.services.AbfsClient.LOG;
+
+/**
+ * AbfsMetricsManager is responsible for managing metrics collection
+ * and emission for an AbfsClient instance.
+ */
+public class AbfsMetricsManager implements Closeable {
+
+ // Timer thread name for AbfsMetricsManager
+ public static final String ABFS_CLIENT_TIMER_THREAD_NAME
+ = "abfs-timer-client";
+
+ // Timer for scheduling metric emission tasks based on idle time
+ private Timer timer;
+
+ // URL for sending metrics
+ private URL metricUrl;
+
+ // Shared key credentials for metric account
+ private SharedKeyCredentials metricSharedkeyCredentials = null;
+
+ // Currently running TimerTask
+ private TimerTask runningTimerTask;
+
+ // Metric analysis periods
+ private final int metricAnalysisPeriod;
+
+ // Metric idle period
+ private final int metricIdlePeriod;
+
+ // Flag to indicate if a separate metric account is used
+ private boolean hasSeparateMetricAccount = false;
+
+ // Flag to indicate if metric collection is enabled
+ private final AtomicBoolean isMetricCollectionEnabled
+ = new AtomicBoolean(false);
+
+ // Metric format for metrics
+ private MetricFormat metricFormat;
+
+ // Flag to indicate if metric collection is stopped
+ private final AtomicBoolean isMetricCollectionStopped;
+
+ // AggregateMetricsManager instance
+ private final AggregateMetricsManager aggregateMetricsManager;
+
+ // Scheduler to emit aggregated metric based on time
+ private ScheduledExecutorService metricsEmitScheduler = null;
+
+ // AbfsConfiguration instance
+ private final AbfsConfiguration abfsConfiguration;
+
+ // AbfsCounters instance
+ private final AbfsCounters abfsCounters;
+
+ // File system ID
+ private final String fileSystemId;
+
+ // Storage account name
+ private final String accountName;
+
+ /**
+ * Constructor for AbfsMetricsManager.
+ *
+ * @param abfsConfiguration AbfsConfiguration object.
+ * @param abfsCounters AbfsCounters object.
+ * @param baseUrlString Base URL string of the AbfsClient.
+ * @param indexLastForwardSlash Index of last forward slash in the base URL string.
+ * @param accountName Storage account name.
+ * @param fileSystemId File system ID.
+ */
+ public AbfsMetricsManager(final AbfsConfiguration abfsConfiguration,
+ final AbfsCounters abfsCounters, final String baseUrlString,
+ final int indexLastForwardSlash, final String accountName,
+ final String fileSystemId) {
+ this.abfsConfiguration = abfsConfiguration;
+ this.abfsCounters = abfsCounters;
+ this.fileSystemId = fileSystemId;
+ this.isMetricCollectionEnabled.set(
+ abfsConfiguration.isMetricsCollectionEnabled());
+ this.isMetricCollectionStopped = new AtomicBoolean(false);
+ this.aggregateMetricsManager = AggregateMetricsManager.getInstance(
+ abfsConfiguration.getMetricsEmitIntervalInMins(),
+ abfsConfiguration.getMaxMetricsCallsPerSecond());
+ this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
+ this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
+ this.accountName = accountName;
+ if (isMetricCollectionEnabled()) {
+ try {
+ String metricAccountName = abfsConfiguration.getMetricAccount();
+ String metricAccountKey = abfsConfiguration.getMetricAccountKey();
+ this.metricFormat = abfsConfiguration.getMetricFormat();
+ if (isNotEmpty(metricAccountName) && isNotEmpty(
+ metricAccountKey)) {
+ int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ metricAccountName + " - account name is not fully qualified.");
+ }
+ try {
+ metricSharedkeyCredentials = new SharedKeyCredentials(
+ metricAccountName.substring(0, dotIndex),
+ metricAccountKey);
+ hasSeparateMetricAccount = true;
+ setMetricsUrl(metricAccountName.startsWith(HTTPS_SCHEME)
+ ? metricAccountName : HTTPS_SCHEME + COLON
+ + FORWARD_SLASH + FORWARD_SLASH + metricAccountName);
+ } catch (IllegalArgumentException e) {
+ throw new IOException(
+ "Exception while initializing metric credentials ", e);
+ }
+ } else {
+ setMetricsUrl(baseUrlString.substring(0, indexLastForwardSlash + 1));
+ }
+ // Once the metric URL is set, initialize the metrics
+ abfsCounters.initializeMetrics(metricFormat, abfsConfiguration);
+ // Metrics emitter scheduler
+ this.metricsEmitScheduler
+ = Executors.newSingleThreadScheduledExecutor();
+ // run every 1 minute to check the metrics count
+ this.metricsEmitScheduler.scheduleWithFixedDelay(
+ () -> {
+ if (abfsCounters.getAbfsBackoffMetrics()
+ .getMetricValue(TOTAL_NUMBER_OF_REQUESTS)
+ >= abfsConfiguration.getMetricsEmitThreshold()) {
+ emitCollectedMetrics();
+ }
+ },
+ abfsConfiguration.getMetricsEmitThresholdIntervalInSecs(),
+ abfsConfiguration.getMetricsEmitThresholdIntervalInSecs(),
+ TimeUnit.SECONDS);
+
+ // run every metricInterval minutes
+ this.metricsEmitScheduler.scheduleWithFixedDelay(
+ this::emitCollectedMetrics,
+ abfsConfiguration.getMetricsEmitIntervalInMins(),
+ abfsConfiguration.getMetricsEmitIntervalInMins(),
+ TimeUnit.MINUTES);
+
+ // emit metrics based on idea time
+ if (abfsConfiguration.shouldEmitMetricsOnIdleTime()) {
+ this.timer = new Timer(
+ ABFS_CLIENT_TIMER_THREAD_NAME, true);
+ timer.schedule(new TimerTaskImpl(),
+ metricIdlePeriod,
+ metricIdlePeriod);
+ }
+ } catch (Exception e) {
+ LOG.error("Metrics disabled. Failed to initialize metrics for {}",
+ baseUrlString, e);
+ this.isMetricCollectionEnabled.set(false);
+ }
+ }
+ }
+
+ /**
+ * Closes the metrics resources.
+ * This method cancels any running timer tasks, shuts down the metrics emission scheduler,
+ * and emits any collected metrics before closing.
+ */
+ @Override
+ public void close() {
+ if (runningTimerTask != null) {
+ runningTimerTask.cancel();
+ }
+ if (timer != null) {
Review Comment:
timer.purge is redundant after timer.cancel
Will add timer = null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2657353150
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingHeaderFormat.java:
##########
@@ -23,6 +23,9 @@ public enum TracingHeaderFormat {
TWO_ID_FORMAT, // <correlation-id>:<client-req-id>
- ALL_ID_FORMAT; // <client-correlation-id>:<client-req-id>:<filesystem-id>
+ ALL_ID_FORMAT, // <client-correlation-id>:<client-req-id>:<filesystem-id>
// :<primary-req-id>:<stream-id>:<hdfs-operation>:<retry-count>
+
+ AGGREGATED_METRICS_FORMAT; // <client-correlation-id>:<filesystem-id>
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3661375043
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 59s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| -1 :x: | javadoc | 0m 45s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 40s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 22s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 17s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 4 new + 5 unchanged - 0 fixed = 9 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| -1 :x: | javadoc | 0m 32s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 31s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 20s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | shadedclient | 26m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 106m 16s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Switch statement found in org.apache.hadoop.fs.azurebfs.AbfsCountersImpl.initializeMetrics(MetricFormat) where one case falls through to the next case At AbfsCountersImpl.java:where one case falls through to the next case At AbfsCountersImpl.java:[lines 210-212] |
| | org.apache.hadoop.fs.azurebfs.services.MetricsBucket.sendMetrics(AbfsClient, ConcurrentLinkedQueue) might ignore java.io.IOException At MetricsBucket.java: At MetricsBucket.java:[line 140] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux d2b486fdd6df 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9111f5eb06a6d5d0e5c77aa1c5d397d0f912b4cf |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/testReport/ |
| Max. process+thread count | 640 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/2/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3665138731
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 2m 0s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| -1 :x: | javadoc | 0m 45s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 39s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 23s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 10s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 21s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| -1 :x: | javadoc | 0m 31s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 26s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | shadedclient | 26m 16s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 106m 21s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Switch statement found in org.apache.hadoop.fs.azurebfs.AbfsCountersImpl.initializeMetrics(MetricFormat, AbfsConfiguration) where one case falls through to the next case At AbfsCountersImpl.java:AbfsConfiguration) where one case falls through to the next case At AbfsCountersImpl.java:[lines 212-214] |
| | org.apache.hadoop.fs.azurebfs.services.MetricsBucket.sendMetrics(AbfsClient, ConcurrentLinkedQueue) might ignore java.io.IOException At MetricsBucket.java: At MetricsBucket.java:[line 140] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 7da15429ed19 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e6a60874bc31fc39d32e2ce128542050c4d4e704 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/testReport/ |
| Max. process+thread count | 790 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/4/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647782731
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
+ public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
Review Comment:
metric account name, key and format have "metric", other configurations have "metrics". I have kept it intentionally. Do you want to change this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650647488
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
+ // This client is the last one — drain metrics now
+ batchToSend = metricsBuffer.getAndSet(new ConcurrentLinkedQueue<>());
+ isLastClient = true;
+ }
+
+ clients.remove(client);
+ }
+ if (isLastClient) {
+ sendMetrics(client, batchToSend);
+ }
+ return true;
+ }
+
+ /**
+ * Add a metric to the buffer.
+ * @param metric the metric to add
+ */
+ void addRequest(String metric) {
+ if (metric != null) {
+ metricsBuffer.get().add(metric);
+ }
+ }
+
+ /**
+ * Drain the metrics buffer and send if there are registered clients.
+ */
+ public void drainAndSendIfReady() {
+ AbfsClient client;
+
+ synchronized (this) {
+ if (clients.isEmpty()) {
+ return;
+ }
+ client = clients.iterator().next();
+ }
+
+ ConcurrentLinkedQueue<String> batch = metricsBuffer.getAndSet(
+ new ConcurrentLinkedQueue<>());
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ sendMetrics(client, batch);
+ }
+
+ // Send metrics outside synchronized block
+ private void sendMetrics(AbfsClient client,
+ ConcurrentLinkedQueue<String> batchToSend) {
+ // Send outside synchronized block
+ if (client != null && batchToSend != null && !batchToSend.isEmpty()) {
+ for (String chunk : splitListBySize(batchToSend, MAX_HEADER_SIZE)) {
+ rateLimiter.acquire(); // Rate limiting
+ try {
+ client.getMetricCall(chunk);
+ } catch (IOException ignored) {}
Review Comment:
Can add a debug line here instead of ignoring exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650743802
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -876,6 +894,308 @@ public void testIsNonEmptyDirectory() throws IOException {
false, 1, true);
}
+ /**
+ * Test to verify that in case metric account is not set,
+ * metric collection is enabled with default metric format
+ * and account url.
+ *
+ * @throws Exception in case of any failure
+ */
+ @Test
+ public void testMetricAccountFallback() throws Exception {
+ Configuration configuration = getRawConfiguration();
+ configuration.setBoolean(
+ AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION, false);
+ configuration.setBoolean(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, false);
+ configuration.unset(FS_AZURE_METRIC_ACCOUNT_NAME);
+ configuration.unset(FS_AZURE_METRIC_ACCOUNT_KEY);
+ configuration.unset(FS_AZURE_METRIC_FORMAT);
+ configuration.setBoolean(FS_AZURE_ALWAYS_USE_HTTPS, true);
+ final AzureBlobFileSystem fs = getFileSystem(configuration);
+ Assertions.assertThat(
+ fs.getAbfsStore().getAbfsConfiguration().getMetricFormat())
+ .describedAs(
+ "In case metric format is not set, metric format should "
+ + "be defaulted to internal metric format")
+ .isEqualTo(MetricFormat.INTERNAL_METRIC_FORMAT);
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().isMetricCollectionEnabled())
+ .describedAs(
+ "Metric collection should be enabled even if metric account is not set")
+ .isTrue();
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().getAbfsCounters().toString())
+ .describedAs(
+ "AbfsCounters should not contain backoff related metrics "
+ + "as no metric is collected for backoff")
+ .doesNotContain("#BO:");
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().getAbfsCounters().toString())
+ .describedAs(
+ "AbfsCounters should not contain read footer related metrics "
+ + "as no metric is collected for read footer")
+ .doesNotContain("#FO:");
+
+ final URIBuilder uriBuilder = new URIBuilder();
+ uriBuilder.setScheme(FileSystemUriSchemes.HTTPS_SCHEME);
+ uriBuilder.setHost(fs.getUri().getHost());
+ uriBuilder.setPath(FORWARD_SLASH);
+ Assertions.assertThat(fs.getAbfsStore().getClient().getMetricsUrl())
+ .describedAs(
+ "In case metric account is not set, account url should be used")
+ .isEqualTo(
+ UriUtils.changeUrlFromBlobToDfs(uriBuilder.build().toURL()));
+ }
+
+ /**
+ * Test to verify that in case metric format is set to empty,
+ * metric collection is disabled.
+ *
+ * @throws Exception in case of any failure
+ */
+ @Test
+ public void testMetricCollectionWithDifferentMetricFormat() throws Exception {
+ Configuration configuration = getRawConfiguration();
+ // Setting this configuration just to ensure there is only one call during filesystem initialization
+ configuration.setBoolean(
+ AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION, true);
+ configuration.setBoolean(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, false);
+ configuration.unset(FS_AZURE_METRIC_FORMAT);
+ configuration.setEnum(FS_AZURE_METRIC_FORMAT,
+ INTERNAL_BACKOFF_METRIC_FORMAT);
+ final AzureBlobFileSystem fs = getFileSystem(configuration);
+ int totalCalls = 1; // Filesystem initialization call
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().isMetricCollectionEnabled())
+ .describedAs("Metric collection should be enabled by default")
+ .isTrue();
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getAbfsConfiguration().getMetricFormat())
+ .describedAs("Metric format should be as set in configuration")
+ .isEqualTo(INTERNAL_BACKOFF_METRIC_FORMAT);
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().getAbfsCounters().toString())
+ .describedAs(
+ "AbfsCounters should only contains backoff related metrics when "
+ + "metric format is internal backoff metric format")
+ .contains("#BO:");
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().getAbfsCounters().toString())
+ .describedAs(
+ "AbfsCounters should not contains read footer related metrics when "
+ + "metric format is internal backoff metric format")
+ .doesNotContain("#FO:");
+
+ Assertions.assertThat(fs.getAbfsStore()
+ .getClient()
+ .getAbfsCounters()
+ .getAbfsBackoffMetrics()
+ .getMetricValue(
+ AbfsBackoffMetricsEnum.TOTAL_NUMBER_OF_REQUESTS))
+ .describedAs(
+ "Total number of requests should be 1 for filesystem initialization")
+ .isEqualTo(totalCalls);
+
+
+ if (fs.getAbfsStore().getClient() instanceof AbfsDfsClient) {
+ intercept(FileNotFoundException.class,
+ "The specified path does not exist.",
+ () -> fs.listStatus(path("/testPath")));
+ totalCalls += 1; // listStatus call
+ } else {
+ intercept(FileNotFoundException.class,
+ "The specified blob does not exist.",
+ () -> fs.listStatus(path("/testPath")));
+ totalCalls += 2; // listStatus call makes 2 calls to the service
+ }
+
+ Assertions.assertThat(fs.getAbfsStore()
+ .getClient()
+ .getAbfsCounters()
+ .getAbfsBackoffMetrics()
+ .getMetricValue(
+ AbfsBackoffMetricsEnum.TOTAL_NUMBER_OF_REQUESTS))
+ .describedAs(
+ "Total number of requests should be 2 after listStatus")
+ .isEqualTo(totalCalls);
+ }
+
+ /**
+ * Test to verify that clientRequestId contains backoff metrics
+ * when metric format is set to internal backoff metric format.
+ *
+ * @throws Exception in case of any failure
+ */
+ @Test
+ public void testGetMetricsCallMethod() throws Exception {
+ // File system init will make few calls to the service.
+ // Backoff metrics will be collected for those calls.
+ AzureBlobFileSystem fs = getFileSystem();
+ TracingContext tracingContext = new TracingContext(
+ fs.getAbfsStore().getAbfsConfiguration().getClientCorrelationId(),
+ "test-filesystem-id", FSOperationType.TEST_OP, true,
+ TracingHeaderFormat.AGGREGATED_METRICS_FORMAT, null,
+ fs.getAbfsStore().getClient().getAbfsCounters().toString());
+
+ AbfsHttpOperation abfsHttpOperation = getAbfsClient(
+ fs.getAbfsStore()).getAbfsRestOperation(
+ AbfsRestOperationType.GetFileSystemProperties,
+ HTTP_METHOD_HEAD,
+ fs.getAbfsStore().getClient().getMetricsUrl(),
+ getTestRequestHeaders(fs.getAbfsStore().getClient()))
+ .createHttpOperation();
+ tracingContext.constructHeader(abfsHttpOperation, null,
+ EXPONENTIAL_RETRY_POLICY_ABBREVIATION);
+ assertThat(abfsHttpOperation.getClientRequestId())
+ .describedAs("ClientRequestId should be contains Backoff metrics")
Review Comment:
nit: should be containing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652910001
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/SimpleRateLimiter.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+
+public final class SimpleRateLimiter {
Review Comment:
nit: we can add some class definition here and if its local to JVM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647763426
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBackoffMetrics.java:
##########
@@ -75,18 +74,24 @@
* retry operations in Azure Blob File System (ABFS).
*/
public class AbfsBackoffMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsBackoffMetrics.class);
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsBackoffMetrics.class);
+
private static final List<RetryValue> RETRY_LIST = Arrays.asList(
- RetryValue.values());
+ RetryValue.values());
+
+ private final boolean isRetryMetricEnabled;
Review Comment:
Yes, it was discussed and agreed upon that we will put retry metrics behind the config with default value false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2643002092
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
Review Comment:
break missing here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2643003666
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -375,8 +394,7 @@ public DurationTracker trackDuration(String key) {
public String toString() {
String metric = "";
Review Comment:
Use empty string constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647826443
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBackoffMetrics.java:
##########
@@ -240,25 +251,24 @@ private String getPrecisionMetrics(AbfsBackoffMetricsEnum metricName,
private void getRetryMetrics(StringBuilder metricBuilder) {
for (RetryValue retryCount : RETRY_LIST) {
long totalRequests = getMetricValue(TOTAL_REQUESTS, retryCount);
- metricBuilder.append(REQUEST_COUNT)
- .append(retryCount.getValue())
- .append(REQUESTS)
- .append(getMetricValue(NUMBER_OF_REQUESTS_SUCCEEDED, retryCount));
+ if (getMetricValue(NUMBER_OF_REQUESTS_SUCCEEDED, retryCount) > 0) {
+ metricBuilder.append(REQUEST_COUNT)
+ .append(retryCount.getValue())
+ .append(REQUESTS)
+ .append(getMetricValue(NUMBER_OF_REQUESTS_SUCCEEDED, retryCount));
+ }
if (totalRequests > 0) {
metricBuilder.append(MIN_MAX_AVERAGE)
- .append(retryCount.getValue())
- .append(REQUESTS)
- .append(getPrecisionMetrics(MIN_BACK_OFF, retryCount, THOUSAND))
- .append(SECONDS)
- .append(getPrecisionMetrics(MAX_BACK_OFF, retryCount, THOUSAND))
- .append(SECONDS)
- .append(getPrecisionMetrics(TOTAL_BACK_OFF, retryCount, totalRequests * THOUSAND))
- .append(SECONDS);
- } else {
- metricBuilder.append(MIN_MAX_AVERAGE)
Review Comment:
Why have we removed the MMA ones ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647814546
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
Review Comment:
Since GetFileSystemProperties is on the root path and this path can be derived directly from account name, so separate config for URI is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2672219586
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleWithFixedDelay(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
+ }));
+ }
+
+ /**
+ * Get the singleton instance of AggregateMetricsManager.
+ *
+ * @param dispatchIntervalInMins Interval in minutes for dispatching metrics.
+ * @param permitsPerSecond Rate limit for dispatching metrics.
+ * @return Singleton instance of AggregateMetricsManager.
+ */
+ public static AggregateMetricsManager getInstance(final long dispatchIntervalInMins,
+ final int permitsPerSecond) {
+ if (instance != null) {
+ return instance;
+ }
+
+ synchronized (AggregateMetricsManager.class) {
+ if (instance == null) {
+ try {
+ instance = new AggregateMetricsManager(
+ dispatchIntervalInMins, permitsPerSecond);
+ } catch (InvalidConfigurationValueException e) {
+ throw new RuntimeException(
+ "Failed to initialize AggregateMetricsManager", e);
+ }
+ }
+ return instance;
+ }
+ }
+
+ /**
+ * Register an AbfsClient with the manager.
+ * @param account Account name.
+ * @param abfsClient AbfsClient instance.
+ */
+ public void registerClient(String account, AbfsClient abfsClient) {
+ if (StringUtils.isEmpty(account) || abfsClient == null) {
+ return;
+ }
+
+ buckets.computeIfAbsent(account,
+ key -> new MetricsBucket(rateLimiter))
+ .registerClient(abfsClient);
+ }
+
+ /**
+ * Deregister an AbfsClient from the manager.
+ * @param account Account name.
+ * @param abfsClient AbfsClient instance.
+ * @return true if the client was deregistered, false otherwise.
+ */
+ public boolean deregisterClient(String account, AbfsClient abfsClient) {
+ if (StringUtils.isEmpty(account) || abfsClient == null) {
+ return false;
+ }
+
+ AtomicBoolean isRemoved = new AtomicBoolean(false);
+
+ buckets.computeIfPresent(account, (key, bucket) -> {
+ // Deregister the client
+ isRemoved.set(bucket.deregisterClient(abfsClient));
+
+ // If bucket became empty, remove it atomically
+ return bucket.isEmpty() ? null : bucket;
+ });
+
+ return isRemoved.get();
+ }
+
+ /**
+ * Record metrics data for a specific account.
+ * @param accountName Account name.
+ * @param metricsData Metrics data to record.
+ */
+ public void recordMetric(String accountName, String metricsData) {
+ if (StringUtils.isEmpty(accountName)
Review Comment:
StringUtils.isEmpty() does both null and empty string check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2665153908
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/SimpleRateLimiter.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+
+public final class SimpleRateLimiter {
+
+ // Interval between permits in nanoseconds.
+ private final long intervalNanos;
+
+ // Next allowed time to acquire a permit in nanoseconds.
+ private long nextAllowedTime;
Review Comment:
It is an overhead if we make this variable Atomic along with synchronized method where it is getting used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2667153513
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
+ private static final String FOOTER_LENGTH = "20";
+
+ private static final List<FileType> FILE_TYPE_LIST =
+ Arrays.asList(FileType.values());
+
+ private Map<String, FileTypeMetrics> fileTypeMetricsMap =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Inner class to handle file type checks.
+ */
+ private static final class FileTypeMetrics {
+
+ private final AtomicBoolean collectMetrics;
+
+ private final AtomicBoolean collectMetricsForNextRead;
+
+ private final AtomicBoolean collectLenMetrics;
+
+ private final AtomicLong readCount;
+
+ private final AtomicLong offsetOfFirstRead;
+
+ private FileType fileType = null;
+
+ private String sizeReadByFirstRead;
+
+ private String offsetDiffBetweenFirstAndSecondRead;
/**
- * Inner class to handle file type checks.
+ * Constructor to initialize the file type metrics.
*/
- private static final class FileTypeMetrics {
- private final AtomicBoolean collectMetrics;
- private final AtomicBoolean collectMetricsForNextRead;
- private final AtomicBoolean collectLenMetrics;
- private final AtomicLong readCount;
- private final AtomicLong offsetOfFirstRead;
- private FileType fileType = null;
- private String sizeReadByFirstRead;
- private String offsetDiffBetweenFirstAndSecondRead;
-
- /**
- * Constructor to initialize the file type metrics.
- */
- private FileTypeMetrics() {
- collectMetrics = new AtomicBoolean(false);
- collectMetricsForNextRead = new AtomicBoolean(false);
- collectLenMetrics = new AtomicBoolean(false);
- readCount = new AtomicLong(0);
- offsetOfFirstRead = new AtomicLong(0);
- }
-
- /**
- * Updates the file type based on the metrics collected.
- */
- private void updateFileType() {
- if (fileType == null) {
- fileType = collectMetrics.get() && readCount.get() >= 2
- && haveEqualValues(sizeReadByFirstRead)
- && haveEqualValues(offsetDiffBetweenFirstAndSecondRead) ? PARQUET : NON_PARQUET;
- }
- }
-
- /**
- * Checks if the given value has equal parts.
- *
- * @param value the value to check
- * @return true if the value has equal parts, false otherwise
- */
- private boolean haveEqualValues(String value) {
- String[] parts = value.split("_");
- return parts.length == 2
- && parts[0].equals(parts[1]);
- }
-
- /**
- * Increments the read count.
- */
- private void incrementReadCount() {
- readCount.incrementAndGet();
- }
-
- /**
- * Returns the read count.
- *
- * @return the read count
- */
- private long getReadCount() {
- return readCount.get();
- }
-
- /**
- * Sets the collect metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetrics(boolean collect) {
- collectMetrics.set(collect);
- }
-
- /**
- * Returns the collect metrics flag.
- *
- * @return the collect metrics flag
- */
- private boolean getCollectMetrics() {
- return collectMetrics.get();
- }
-
- /**
- * Sets the collect metrics for the next read flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetricsForNextRead(boolean collect) {
- collectMetricsForNextRead.set(collect);
- }
-
- /**
- * Returns the collect metrics for the next read flag.
- *
- * @return the collect metrics for the next read flag
- */
- private boolean getCollectMetricsForNextRead() {
- return collectMetricsForNextRead.get();
- }
-
- /**
- * Returns the collect length metrics flag.
- *
- * @return the collect length metrics flag
- */
- private boolean getCollectLenMetrics() {
- return collectLenMetrics.get();
- }
-
- /**
- * Sets the collect length metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectLenMetrics(boolean collect) {
- collectLenMetrics.set(collect);
- }
-
- /**
- * Sets the offset of the first read.
- *
- * @param offset the value to set
- */
- private void setOffsetOfFirstRead(long offset) {
- offsetOfFirstRead.set(offset);
- }
-
- /**
- * Returns the offset of the first read.
- *
- * @return the offset of the first read
- */
- private long getOffsetOfFirstRead() {
- return offsetOfFirstRead.get();
- }
-
- /**
- * Sets the size read by the first read.
- *
- * @param size the value to set
- */
- private void setSizeReadByFirstRead(String size) {
- sizeReadByFirstRead = size;
- }
-
- /**
- * Returns the size read by the first read.
- *
- * @return the size read by the first read
- */
- private String getSizeReadByFirstRead() {
- return sizeReadByFirstRead;
- }
-
- /**
- * Sets the offset difference between the first and second read.
- *
- * @param offsetDiff the value to set
- */
- private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
- offsetDiffBetweenFirstAndSecondRead = offsetDiff;
- }
-
- /**
- * Returns the offset difference between the first and second read.
- *
- * @return the offset difference between the first and second read
- */
- private String getOffsetDiffBetweenFirstAndSecondRead() {
- return offsetDiffBetweenFirstAndSecondRead;
- }
-
- /**
- * Returns the file type.
- *
- * @return the file type
- */
- private FileType getFileType() {
- return fileType;
- }
+ private FileTypeMetrics() {
+ collectMetrics = new AtomicBoolean(false);
+ collectMetricsForNextRead = new AtomicBoolean(false);
+ collectLenMetrics = new AtomicBoolean(false);
+ readCount = new AtomicLong(0);
+ offsetOfFirstRead = new AtomicLong(0);
}
/**
- * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ * Updates the file type based on the metrics collected.
*/
- public AbfsReadFooterMetrics() {
- IOStatisticsStore ioStatisticsStore = iostatisticsStore()
- .withCounters(getMetricNames(TYPE_COUNTER))
- .withMeanStatistics(getMetricNames(TYPE_MEAN))
- .build();
- setIOStatistics(ioStatisticsStore);
+ private void updateFileType() {
+ if (fileType == null) {
+ fileType = collectMetrics.get() && readCount.get() >= 2
+ && haveEqualValues(sizeReadByFirstRead)
+ && haveEqualValues(offsetDiffBetweenFirstAndSecondRead)
+ ? PARQUET
+ : NON_PARQUET;
+ }
}
/**
- * Returns the metric names for a specific statistic type.
+ * Checks if the given value has equal parts.
*
- * @param type the statistic type
- * @return the metric names
+ * @param value the value to check
+ * @return true if the value has equal parts, false otherwise
*/
- private String[] getMetricNames(StatisticTypeEnum type) {
- return Arrays.stream(AbfsReadFooterMetricsEnum.values())
- .filter(readFooterMetricsEnum -> readFooterMetricsEnum.getStatisticType().equals(type))
- .flatMap(readFooterMetricsEnum ->
- FILE.equals(readFooterMetricsEnum.getType())
- ? FILE_TYPE_LIST.stream().map(fileType ->
- getMetricName(fileType, readFooterMetricsEnum))
- : Stream.of(readFooterMetricsEnum.getName()))
- .toArray(String[]::new);
+ private boolean haveEqualValues(String value) {
+ String[] parts = value.split("_");
+ return parts.length == 2
+ && parts[0].equals(parts[1]);
}
/**
- * Returns the metric name for a specific file type and metric.
- *
- * @param fileType the type of the file
- * @param readFooterMetricsEnum the metric to get the name for
- * @return the metric name
+ * Increments the read count.
*/
- private String getMetricName(FileType fileType,
- AbfsReadFooterMetricsEnum readFooterMetricsEnum) {
- if (fileType == null || readFooterMetricsEnum == null) {
- LOG.error("File type or ABFS read footer metrics should not be null");
- return EMPTY_STRING;
- }
- return fileType + COLON + readFooterMetricsEnum.getName();
+ private void incrementReadCount() {
+ readCount.incrementAndGet();
}
/**
- * Looks up the counter value for a specific metric.
+ * Returns the read count.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to look up
- * @return the counter value
+ * @return the read count
*/
- private long getCounterMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- return lookupCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ private long getReadCount() {
+ return readCount.get();
}
/**
- * Looks up the mean statistic value for a specific metric.
+ * Sets the collect metrics flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to look up
- * @return the mean statistic value
+ * @param collect the value to set
*/
- private String getMeanMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- return format(DOUBLE_PRECISION_FORMAT,
- lookupMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum)));
+ private void setCollectMetrics(boolean collect) {
+ collectMetrics.set(collect);
}
/**
- * Increments the value of a specific metric.
+ * Returns the collect metrics flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to increment
+ * @return the collect metrics flag
*/
- public void incrementMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- incCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ private boolean getCollectMetrics() {
+ return collectMetrics.get();
}
/**
- * Adds a mean statistic value for a specific metric.
+ * Sets the collect metrics for the next read flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to update
- * @param value the new value of the metric
+ * @param collect the value to set
*/
- public void addMeanMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum,
- long value) {
- addMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum), value);
+ private void setCollectMetricsForNextRead(boolean collect) {
+ collectMetricsForNextRead.set(collect);
}
/**
- * Returns the total number of files.
+ * Returns the collect metrics for the next read flag.
*
- * @return the total number of files
+ * @return the collect metrics for the next read flag
*/
- public Long getTotalFiles() {
- return getCounterMetricValue(PARQUET, TOTAL_FILES) + getCounterMetricValue(NON_PARQUET, TOTAL_FILES);
+ private boolean getCollectMetricsForNextRead() {
+ return collectMetricsForNextRead.get();
}
/**
- * Updates the map with a new file path identifier.
+ * Returns the collect length metrics flag.
*
- * @param filePathIdentifier the file path identifier
+ * @return the collect length metrics flag
*/
- public void updateMap(String filePathIdentifier) {
- fileTypeMetricsMap.computeIfAbsent(filePathIdentifier, key -> new FileTypeMetrics());
+ private boolean getCollectLenMetrics() {
+ return collectLenMetrics.get();
}
/**
- * Checks and updates the metrics for a given file read.
+ * Sets the collect length metrics flag.
*
- * @param filePathIdentifier the file path identifier
- * @param len the length of the read
- * @param contentLength the total content length of the file
- * @param nextReadPos the position of the next read
+ * @param collect the value to set
*/
- public void updateReadMetrics(final String filePathIdentifier,
- final int len,
- final long contentLength,
- final long nextReadPos) {
- FileTypeMetrics fileTypeMetrics = fileTypeMetricsMap.computeIfAbsent(filePathIdentifier, key -> new FileTypeMetrics());
- if (fileTypeMetrics.getReadCount() == 0 || (fileTypeMetrics.getReadCount() >= 1 && fileTypeMetrics.getCollectMetrics())) {
- updateMetrics(fileTypeMetrics, len, contentLength, nextReadPos);
- }
+ private void setCollectLenMetrics(boolean collect) {
+ collectLenMetrics.set(collect);
}
/**
- * Updates metrics for a specific file identified by filePathIdentifier.
+ * Sets the offset of the first read.
*
- * @param fileTypeMetrics File metadata to know file type.
- * @param len The length of the read operation.
- * @param contentLength The total content length of the file.
- * @param nextReadPos The position of the next read operation.
+ * @param offset the value to set
*/
- private void updateMetrics(FileTypeMetrics fileTypeMetrics,
- int len,
- long contentLength,
- long nextReadPos) {
- fileTypeMetrics.incrementReadCount();
-
- long readCount = fileTypeMetrics.getReadCount();
-
- if (readCount == 1) {
- handleFirstRead(fileTypeMetrics, nextReadPos, len, contentLength);
- } else if (readCount == 2) {
- handleSecondRead(fileTypeMetrics, nextReadPos, len, contentLength);
- } else {
- handleFurtherRead(fileTypeMetrics, len);
- }
+ private void setOffsetOfFirstRead(long offset) {
+ offsetOfFirstRead.set(offset);
}
/**
- * Handles the first read operation by checking if the current read position is near the end of the file.
- * If it is, updates the {@link FileTypeMetrics} object to enable metrics collection and records the first read's
- * offset and size.
+ * Returns the offset of the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
- * @param nextReadPos The position where the next read will start.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @return the offset of the first read
*/
- private void handleFirstRead(FileTypeMetrics fileTypeMetrics,
- long nextReadPos,
- int len,
- long contentLength) {
- if (nextReadPos >= contentLength - (long) Integer.parseInt(FOOTER_LENGTH) * ONE_KB) {
- fileTypeMetrics.setCollectMetrics(true);
- fileTypeMetrics.setCollectMetricsForNextRead(true);
- fileTypeMetrics.setOffsetOfFirstRead(nextReadPos);
- fileTypeMetrics.setSizeReadByFirstRead(len + "_" + Math.abs(contentLength - nextReadPos));
- }
+ private long getOffsetOfFirstRead() {
+ return offsetOfFirstRead.get();
}
/**
- * Handles the second read operation by checking if metrics collection is enabled for the next read.
- * If it is, calculates the offset difference between the first and second reads, updates the {@link FileTypeMetrics}
- * object with this information, and sets the file type. Then, updates the metrics data.
+ * Sets the size read by the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
- * @param nextReadPos The position where the next read will start.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @param size the value to set
*/
- private void handleSecondRead(FileTypeMetrics fileTypeMetrics,
- long nextReadPos,
- int len,
- long contentLength) {
- if (fileTypeMetrics.getCollectMetricsForNextRead()) {
- long offsetDiff = Math.abs(nextReadPos - fileTypeMetrics.getOffsetOfFirstRead());
- fileTypeMetrics.setOffsetDiffBetweenFirstAndSecondRead(len + "_" + offsetDiff);
- fileTypeMetrics.setCollectLenMetrics(true);
- fileTypeMetrics.updateFileType();
- updateMetricsData(fileTypeMetrics, len, contentLength);
- }
+ private void setSizeReadByFirstRead(String size) {
+ sizeReadByFirstRead = size;
}
/**
- * Handles further read operations beyond the second read. If metrics collection is enabled and the file type is set,
- * updates the read length requested and increments the read count for the specific file type.
+ * Returns the size read by the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object containing metrics and read details.
- * @param len The length of the current read operation.
+ * @return the size read by the first read
*/
- private void handleFurtherRead(FileTypeMetrics fileTypeMetrics, int len) {
- if (fileTypeMetrics.getCollectLenMetrics() && fileTypeMetrics.getFileType() != null) {
- FileType fileType = fileTypeMetrics.getFileType();
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, len);
- }
+ private String getSizeReadByFirstRead() {
+ return sizeReadByFirstRead;
}
/**
- * Updates the metrics data for a specific file identified by the {@link FileTypeMetrics} object.
- * This method calculates and updates various metrics such as read length requested, file length,
- * size read by the first read, and offset differences between reads.
+ * Sets the offset difference between the first and second read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object containing metrics and read details.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @param offsetDiff the value to set
*/
- private void updateMetricsData(FileTypeMetrics fileTypeMetrics,
- int len,
- long contentLength) {
- long sizeReadByFirstRead = Long.parseLong(fileTypeMetrics.getSizeReadByFirstRead().split("_")[0]);
- long firstOffsetDiff = Long.parseLong(fileTypeMetrics.getSizeReadByFirstRead().split("_")[1]);
- long secondOffsetDiff = Long.parseLong(fileTypeMetrics.getOffsetDiffBetweenFirstAndSecondRead().split("_")[1]);
- FileType fileType = fileTypeMetrics.getFileType();
-
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, len);
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, sizeReadByFirstRead);
- addMeanMetricValue(fileType, AVG_FILE_LENGTH, contentLength);
- addMeanMetricValue(fileType, AVG_SIZE_READ_BY_FIRST_READ, sizeReadByFirstRead);
- addMeanMetricValue(fileType, AVG_OFFSET_DIFF_BETWEEN_FIRST_AND_SECOND_READ, len);
- addMeanMetricValue(fileType, AVG_FIRST_OFFSET_DIFF, firstOffsetDiff);
- addMeanMetricValue(fileType, AVG_SECOND_OFFSET_DIFF, secondOffsetDiff);
- incrementMetricValue(fileType, TOTAL_FILES);
+ private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
+ offsetDiffBetweenFirstAndSecondRead = offsetDiff;
}
/**
- * Appends the metrics for a specific file type to the given metric builder.
+ * Returns the offset difference between the first and second read.
*
- * @param metricBuilder the metric builder to append the metrics to
- * @param fileType the file type to append the metrics for
+ * @return the offset difference between the first and second read
*/
- private void appendMetrics(StringBuilder metricBuilder, FileType fileType) {
- long totalFiles = getCounterMetricValue(fileType, TOTAL_FILES);
- if (totalFiles <= 0) {
- return;
- }
-
- String sizeReadByFirstRead = getMeanMetricValue(fileType, AVG_SIZE_READ_BY_FIRST_READ);
- String offsetDiffBetweenFirstAndSecondRead = getMeanMetricValue(fileType, AVG_OFFSET_DIFF_BETWEEN_FIRST_AND_SECOND_READ);
-
- if (NON_PARQUET.equals(fileType)) {
- sizeReadByFirstRead += CHAR_UNDERSCORE + getMeanMetricValue(fileType, AVG_FIRST_OFFSET_DIFF);
- offsetDiffBetweenFirstAndSecondRead += CHAR_UNDERSCORE + getMeanMetricValue(fileType, AVG_SECOND_OFFSET_DIFF);
- }
-
- metricBuilder.append(CHAR_DOLLAR)
- .append(fileType)
- .append(FIRST_READ)
- .append(sizeReadByFirstRead)
- .append(SECOND_READ)
- .append(offsetDiffBetweenFirstAndSecondRead)
- .append(FILE_LENGTH)
- .append(getMeanMetricValue(fileType, AVG_FILE_LENGTH))
- .append(READ_LENGTH)
- .append(getMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED));
+ private String getOffsetDiffBetweenFirstAndSecondRead() {
+ return offsetDiffBetweenFirstAndSecondRead;
}
/**
- * Returns the read footer metrics for all file types.
+ * Returns the file type.
*
- * @return the read footer metrics as a string
+ * @return the file type
*/
- @Override
- public String toString() {
- StringBuilder readFooterMetric = new StringBuilder();
- appendMetrics(readFooterMetric, PARQUET);
- appendMetrics(readFooterMetric, NON_PARQUET);
- return readFooterMetric.toString();
+ private FileType getFileType() {
+ return fileType;
+ }
+ }
+
+ /**
+ * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ */
+ public AbfsReadFooterMetrics() {
+ IOStatisticsStore ioStatisticsStore = iostatisticsStore()
+ .withCounters(getMetricNames(TYPE_COUNTER))
+ .withMeanStatistics(getMetricNames(TYPE_MEAN))
+ .build();
+ setIOStatistics(ioStatisticsStore);
+ }
+
+ /**
+ * Constructor to initialize the IOStatisticsStore with counters and mean statistics,
+ * and a map to track file type metrics.
+ *
+ * @param fileTypeMetricsMap the map to track file type metrics
+ */
+ public AbfsReadFooterMetrics(Map<String, FileTypeMetrics> fileTypeMetricsMap) {
Review Comment:
Better to keep these constructors separate otherwise from all other places we will need to pass empty map or null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3718780366
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 15m 29s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 22s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 22s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 24s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 46s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 22s | [/results-checkstyle-hadoop-tools_hadoop-azure.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/artifact/out/results-checkstyle-hadoop-tools_hadoop-azure.txt) | hadoop-tools/hadoop-azure: The patch generated 9 new + 5 unchanged - 0 fixed = 14 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| -1 :x: | javadoc | 0m 32s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 29s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | spotbugs | 1m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 31s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 119m 55s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b0d87e2631ae 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c89af4368e382e7b9e0fd5f8372c22c6e155b25d |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/testReport/ |
| Max. process+thread count | 769 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/6/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3722879786
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 42s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 20s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/8/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 28s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 50s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| -1 :x: | javadoc | 0m 31s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/8/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | spotbugs | 1m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 12s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 20s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not generate ASF License warnings. |
| | | 105m 28s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 60dde8514129 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fbd203fa98e588ba223c3078d7506631d41b9d7b |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/8/testReport/ |
| Max. process+thread count | 641 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/8/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647826443
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBackoffMetrics.java:
##########
@@ -240,25 +251,24 @@ private String getPrecisionMetrics(AbfsBackoffMetricsEnum metricName,
private void getRetryMetrics(StringBuilder metricBuilder) {
for (RetryValue retryCount : RETRY_LIST) {
long totalRequests = getMetricValue(TOTAL_REQUESTS, retryCount);
- metricBuilder.append(REQUEST_COUNT)
- .append(retryCount.getValue())
- .append(REQUESTS)
- .append(getMetricValue(NUMBER_OF_REQUESTS_SUCCEEDED, retryCount));
+ if (getMetricValue(NUMBER_OF_REQUESTS_SUCCEEDED, retryCount) > 0) {
+ metricBuilder.append(REQUEST_COUNT)
+ .append(retryCount.getValue())
+ .append(REQUESTS)
+ .append(getMetricValue(NUMBER_OF_REQUESTS_SUCCEEDED, retryCount));
+ }
if (totalRequests > 0) {
metricBuilder.append(MIN_MAX_AVERAGE)
- .append(retryCount.getValue())
- .append(REQUESTS)
- .append(getPrecisionMetrics(MIN_BACK_OFF, retryCount, THOUSAND))
- .append(SECONDS)
- .append(getPrecisionMetrics(MAX_BACK_OFF, retryCount, THOUSAND))
- .append(SECONDS)
- .append(getPrecisionMetrics(TOTAL_BACK_OFF, retryCount, totalRequests * THOUSAND))
- .append(SECONDS);
- } else {
- metricBuilder.append(MIN_MAX_AVERAGE)
Review Comment:
Why have we removed the MMA ones ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647758193
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##########
@@ -136,6 +136,13 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_ENABLE_AUTOTHROTTLING = false;
public static final int DEFAULT_METRIC_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS = 60_000;
+ public static final boolean DEFAULT_METRICS_COLLECTION_ENABLED = true;
+ public static final boolean DEFAULT_SHOULD_EMIT_METRICS_ON_IDLE_TIME = false;
+ public static final long DEFAULT_METRICS_EMIT_THRESHOLD = 100_000L;
+ public static final long DEFAULT_METRICS_EMIT_THRESHOLD_INTERVAL_SECS = 60;
Review Comment:
Yes, earlier as well we keep it DEFAULT_<part of config name with FS_AZURE>. We can discuss it offline as it will make the variable name bigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650598211
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
+ private static final String FOOTER_LENGTH = "20";
+
+ private static final List<FileType> FILE_TYPE_LIST =
+ Arrays.asList(FileType.values());
+
+ private Map<String, FileTypeMetrics> fileTypeMetricsMap =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Inner class to handle file type checks.
+ */
+ private static final class FileTypeMetrics {
+
+ private final AtomicBoolean collectMetrics;
+
+ private final AtomicBoolean collectMetricsForNextRead;
+
+ private final AtomicBoolean collectLenMetrics;
+
+ private final AtomicLong readCount;
+
+ private final AtomicLong offsetOfFirstRead;
+
+ private FileType fileType = null;
+
+ private String sizeReadByFirstRead;
+
+ private String offsetDiffBetweenFirstAndSecondRead;
/**
- * Inner class to handle file type checks.
+ * Constructor to initialize the file type metrics.
*/
- private static final class FileTypeMetrics {
- private final AtomicBoolean collectMetrics;
- private final AtomicBoolean collectMetricsForNextRead;
- private final AtomicBoolean collectLenMetrics;
- private final AtomicLong readCount;
- private final AtomicLong offsetOfFirstRead;
- private FileType fileType = null;
- private String sizeReadByFirstRead;
- private String offsetDiffBetweenFirstAndSecondRead;
-
- /**
- * Constructor to initialize the file type metrics.
- */
- private FileTypeMetrics() {
- collectMetrics = new AtomicBoolean(false);
- collectMetricsForNextRead = new AtomicBoolean(false);
- collectLenMetrics = new AtomicBoolean(false);
- readCount = new AtomicLong(0);
- offsetOfFirstRead = new AtomicLong(0);
- }
-
- /**
- * Updates the file type based on the metrics collected.
- */
- private void updateFileType() {
- if (fileType == null) {
- fileType = collectMetrics.get() && readCount.get() >= 2
- && haveEqualValues(sizeReadByFirstRead)
- && haveEqualValues(offsetDiffBetweenFirstAndSecondRead) ? PARQUET : NON_PARQUET;
- }
- }
-
- /**
- * Checks if the given value has equal parts.
- *
- * @param value the value to check
- * @return true if the value has equal parts, false otherwise
- */
- private boolean haveEqualValues(String value) {
- String[] parts = value.split("_");
- return parts.length == 2
- && parts[0].equals(parts[1]);
- }
-
- /**
- * Increments the read count.
- */
- private void incrementReadCount() {
- readCount.incrementAndGet();
- }
-
- /**
- * Returns the read count.
- *
- * @return the read count
- */
- private long getReadCount() {
- return readCount.get();
- }
-
- /**
- * Sets the collect metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetrics(boolean collect) {
- collectMetrics.set(collect);
- }
-
- /**
- * Returns the collect metrics flag.
- *
- * @return the collect metrics flag
- */
- private boolean getCollectMetrics() {
- return collectMetrics.get();
- }
-
- /**
- * Sets the collect metrics for the next read flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetricsForNextRead(boolean collect) {
- collectMetricsForNextRead.set(collect);
- }
-
- /**
- * Returns the collect metrics for the next read flag.
- *
- * @return the collect metrics for the next read flag
- */
- private boolean getCollectMetricsForNextRead() {
- return collectMetricsForNextRead.get();
- }
-
- /**
- * Returns the collect length metrics flag.
- *
- * @return the collect length metrics flag
- */
- private boolean getCollectLenMetrics() {
- return collectLenMetrics.get();
- }
-
- /**
- * Sets the collect length metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectLenMetrics(boolean collect) {
- collectLenMetrics.set(collect);
- }
-
- /**
- * Sets the offset of the first read.
- *
- * @param offset the value to set
- */
- private void setOffsetOfFirstRead(long offset) {
- offsetOfFirstRead.set(offset);
- }
-
- /**
- * Returns the offset of the first read.
- *
- * @return the offset of the first read
- */
- private long getOffsetOfFirstRead() {
- return offsetOfFirstRead.get();
- }
-
- /**
- * Sets the size read by the first read.
- *
- * @param size the value to set
- */
- private void setSizeReadByFirstRead(String size) {
- sizeReadByFirstRead = size;
- }
-
- /**
- * Returns the size read by the first read.
- *
- * @return the size read by the first read
- */
- private String getSizeReadByFirstRead() {
- return sizeReadByFirstRead;
- }
-
- /**
- * Sets the offset difference between the first and second read.
- *
- * @param offsetDiff the value to set
- */
- private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
- offsetDiffBetweenFirstAndSecondRead = offsetDiff;
- }
-
- /**
- * Returns the offset difference between the first and second read.
- *
- * @return the offset difference between the first and second read
- */
- private String getOffsetDiffBetweenFirstAndSecondRead() {
- return offsetDiffBetweenFirstAndSecondRead;
- }
-
- /**
- * Returns the file type.
- *
- * @return the file type
- */
- private FileType getFileType() {
- return fileType;
- }
+ private FileTypeMetrics() {
+ collectMetrics = new AtomicBoolean(false);
+ collectMetricsForNextRead = new AtomicBoolean(false);
+ collectLenMetrics = new AtomicBoolean(false);
+ readCount = new AtomicLong(0);
+ offsetOfFirstRead = new AtomicLong(0);
}
/**
- * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ * Updates the file type based on the metrics collected.
*/
- public AbfsReadFooterMetrics() {
- IOStatisticsStore ioStatisticsStore = iostatisticsStore()
- .withCounters(getMetricNames(TYPE_COUNTER))
- .withMeanStatistics(getMetricNames(TYPE_MEAN))
- .build();
- setIOStatistics(ioStatisticsStore);
+ private void updateFileType() {
+ if (fileType == null) {
+ fileType = collectMetrics.get() && readCount.get() >= 2
+ && haveEqualValues(sizeReadByFirstRead)
+ && haveEqualValues(offsetDiffBetweenFirstAndSecondRead)
+ ? PARQUET
+ : NON_PARQUET;
+ }
}
/**
- * Returns the metric names for a specific statistic type.
+ * Checks if the given value has equal parts.
*
- * @param type the statistic type
- * @return the metric names
+ * @param value the value to check
+ * @return true if the value has equal parts, false otherwise
*/
- private String[] getMetricNames(StatisticTypeEnum type) {
- return Arrays.stream(AbfsReadFooterMetricsEnum.values())
- .filter(readFooterMetricsEnum -> readFooterMetricsEnum.getStatisticType().equals(type))
- .flatMap(readFooterMetricsEnum ->
- FILE.equals(readFooterMetricsEnum.getType())
- ? FILE_TYPE_LIST.stream().map(fileType ->
- getMetricName(fileType, readFooterMetricsEnum))
- : Stream.of(readFooterMetricsEnum.getName()))
- .toArray(String[]::new);
+ private boolean haveEqualValues(String value) {
+ String[] parts = value.split("_");
+ return parts.length == 2
+ && parts[0].equals(parts[1]);
}
/**
- * Returns the metric name for a specific file type and metric.
- *
- * @param fileType the type of the file
- * @param readFooterMetricsEnum the metric to get the name for
- * @return the metric name
+ * Increments the read count.
*/
- private String getMetricName(FileType fileType,
- AbfsReadFooterMetricsEnum readFooterMetricsEnum) {
- if (fileType == null || readFooterMetricsEnum == null) {
- LOG.error("File type or ABFS read footer metrics should not be null");
- return EMPTY_STRING;
- }
- return fileType + COLON + readFooterMetricsEnum.getName();
+ private void incrementReadCount() {
+ readCount.incrementAndGet();
}
/**
- * Looks up the counter value for a specific metric.
+ * Returns the read count.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to look up
- * @return the counter value
+ * @return the read count
*/
- private long getCounterMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- return lookupCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ private long getReadCount() {
+ return readCount.get();
}
/**
- * Looks up the mean statistic value for a specific metric.
+ * Sets the collect metrics flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to look up
- * @return the mean statistic value
+ * @param collect the value to set
*/
- private String getMeanMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- return format(DOUBLE_PRECISION_FORMAT,
- lookupMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum)));
+ private void setCollectMetrics(boolean collect) {
+ collectMetrics.set(collect);
}
/**
- * Increments the value of a specific metric.
+ * Returns the collect metrics flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to increment
+ * @return the collect metrics flag
*/
- public void incrementMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- incCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ private boolean getCollectMetrics() {
+ return collectMetrics.get();
}
/**
- * Adds a mean statistic value for a specific metric.
+ * Sets the collect metrics for the next read flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to update
- * @param value the new value of the metric
+ * @param collect the value to set
*/
- public void addMeanMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum,
- long value) {
- addMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum), value);
+ private void setCollectMetricsForNextRead(boolean collect) {
+ collectMetricsForNextRead.set(collect);
}
/**
- * Returns the total number of files.
+ * Returns the collect metrics for the next read flag.
*
- * @return the total number of files
+ * @return the collect metrics for the next read flag
*/
- public Long getTotalFiles() {
- return getCounterMetricValue(PARQUET, TOTAL_FILES) + getCounterMetricValue(NON_PARQUET, TOTAL_FILES);
+ private boolean getCollectMetricsForNextRead() {
+ return collectMetricsForNextRead.get();
}
/**
- * Updates the map with a new file path identifier.
+ * Returns the collect length metrics flag.
*
- * @param filePathIdentifier the file path identifier
+ * @return the collect length metrics flag
*/
- public void updateMap(String filePathIdentifier) {
- fileTypeMetricsMap.computeIfAbsent(filePathIdentifier, key -> new FileTypeMetrics());
+ private boolean getCollectLenMetrics() {
+ return collectLenMetrics.get();
}
/**
- * Checks and updates the metrics for a given file read.
+ * Sets the collect length metrics flag.
*
- * @param filePathIdentifier the file path identifier
- * @param len the length of the read
- * @param contentLength the total content length of the file
- * @param nextReadPos the position of the next read
+ * @param collect the value to set
*/
- public void updateReadMetrics(final String filePathIdentifier,
- final int len,
- final long contentLength,
- final long nextReadPos) {
- FileTypeMetrics fileTypeMetrics = fileTypeMetricsMap.computeIfAbsent(filePathIdentifier, key -> new FileTypeMetrics());
- if (fileTypeMetrics.getReadCount() == 0 || (fileTypeMetrics.getReadCount() >= 1 && fileTypeMetrics.getCollectMetrics())) {
- updateMetrics(fileTypeMetrics, len, contentLength, nextReadPos);
- }
+ private void setCollectLenMetrics(boolean collect) {
+ collectLenMetrics.set(collect);
}
/**
- * Updates metrics for a specific file identified by filePathIdentifier.
+ * Sets the offset of the first read.
*
- * @param fileTypeMetrics File metadata to know file type.
- * @param len The length of the read operation.
- * @param contentLength The total content length of the file.
- * @param nextReadPos The position of the next read operation.
+ * @param offset the value to set
*/
- private void updateMetrics(FileTypeMetrics fileTypeMetrics,
- int len,
- long contentLength,
- long nextReadPos) {
- fileTypeMetrics.incrementReadCount();
-
- long readCount = fileTypeMetrics.getReadCount();
-
- if (readCount == 1) {
- handleFirstRead(fileTypeMetrics, nextReadPos, len, contentLength);
- } else if (readCount == 2) {
- handleSecondRead(fileTypeMetrics, nextReadPos, len, contentLength);
- } else {
- handleFurtherRead(fileTypeMetrics, len);
- }
+ private void setOffsetOfFirstRead(long offset) {
+ offsetOfFirstRead.set(offset);
}
/**
- * Handles the first read operation by checking if the current read position is near the end of the file.
- * If it is, updates the {@link FileTypeMetrics} object to enable metrics collection and records the first read's
- * offset and size.
+ * Returns the offset of the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
- * @param nextReadPos The position where the next read will start.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @return the offset of the first read
*/
- private void handleFirstRead(FileTypeMetrics fileTypeMetrics,
- long nextReadPos,
- int len,
- long contentLength) {
- if (nextReadPos >= contentLength - (long) Integer.parseInt(FOOTER_LENGTH) * ONE_KB) {
- fileTypeMetrics.setCollectMetrics(true);
- fileTypeMetrics.setCollectMetricsForNextRead(true);
- fileTypeMetrics.setOffsetOfFirstRead(nextReadPos);
- fileTypeMetrics.setSizeReadByFirstRead(len + "_" + Math.abs(contentLength - nextReadPos));
- }
+ private long getOffsetOfFirstRead() {
+ return offsetOfFirstRead.get();
}
/**
- * Handles the second read operation by checking if metrics collection is enabled for the next read.
- * If it is, calculates the offset difference between the first and second reads, updates the {@link FileTypeMetrics}
- * object with this information, and sets the file type. Then, updates the metrics data.
+ * Sets the size read by the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
- * @param nextReadPos The position where the next read will start.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @param size the value to set
*/
- private void handleSecondRead(FileTypeMetrics fileTypeMetrics,
- long nextReadPos,
- int len,
- long contentLength) {
- if (fileTypeMetrics.getCollectMetricsForNextRead()) {
- long offsetDiff = Math.abs(nextReadPos - fileTypeMetrics.getOffsetOfFirstRead());
- fileTypeMetrics.setOffsetDiffBetweenFirstAndSecondRead(len + "_" + offsetDiff);
- fileTypeMetrics.setCollectLenMetrics(true);
- fileTypeMetrics.updateFileType();
- updateMetricsData(fileTypeMetrics, len, contentLength);
- }
+ private void setSizeReadByFirstRead(String size) {
+ sizeReadByFirstRead = size;
}
/**
- * Handles further read operations beyond the second read. If metrics collection is enabled and the file type is set,
- * updates the read length requested and increments the read count for the specific file type.
+ * Returns the size read by the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object containing metrics and read details.
- * @param len The length of the current read operation.
+ * @return the size read by the first read
*/
- private void handleFurtherRead(FileTypeMetrics fileTypeMetrics, int len) {
- if (fileTypeMetrics.getCollectLenMetrics() && fileTypeMetrics.getFileType() != null) {
- FileType fileType = fileTypeMetrics.getFileType();
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, len);
- }
+ private String getSizeReadByFirstRead() {
+ return sizeReadByFirstRead;
}
/**
- * Updates the metrics data for a specific file identified by the {@link FileTypeMetrics} object.
- * This method calculates and updates various metrics such as read length requested, file length,
- * size read by the first read, and offset differences between reads.
+ * Sets the offset difference between the first and second read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object containing metrics and read details.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @param offsetDiff the value to set
*/
- private void updateMetricsData(FileTypeMetrics fileTypeMetrics,
- int len,
- long contentLength) {
- long sizeReadByFirstRead = Long.parseLong(fileTypeMetrics.getSizeReadByFirstRead().split("_")[0]);
- long firstOffsetDiff = Long.parseLong(fileTypeMetrics.getSizeReadByFirstRead().split("_")[1]);
- long secondOffsetDiff = Long.parseLong(fileTypeMetrics.getOffsetDiffBetweenFirstAndSecondRead().split("_")[1]);
- FileType fileType = fileTypeMetrics.getFileType();
-
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, len);
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, sizeReadByFirstRead);
- addMeanMetricValue(fileType, AVG_FILE_LENGTH, contentLength);
- addMeanMetricValue(fileType, AVG_SIZE_READ_BY_FIRST_READ, sizeReadByFirstRead);
- addMeanMetricValue(fileType, AVG_OFFSET_DIFF_BETWEEN_FIRST_AND_SECOND_READ, len);
- addMeanMetricValue(fileType, AVG_FIRST_OFFSET_DIFF, firstOffsetDiff);
- addMeanMetricValue(fileType, AVG_SECOND_OFFSET_DIFF, secondOffsetDiff);
- incrementMetricValue(fileType, TOTAL_FILES);
+ private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
+ offsetDiffBetweenFirstAndSecondRead = offsetDiff;
}
/**
- * Appends the metrics for a specific file type to the given metric builder.
+ * Returns the offset difference between the first and second read.
*
- * @param metricBuilder the metric builder to append the metrics to
- * @param fileType the file type to append the metrics for
+ * @return the offset difference between the first and second read
*/
- private void appendMetrics(StringBuilder metricBuilder, FileType fileType) {
- long totalFiles = getCounterMetricValue(fileType, TOTAL_FILES);
- if (totalFiles <= 0) {
- return;
- }
-
- String sizeReadByFirstRead = getMeanMetricValue(fileType, AVG_SIZE_READ_BY_FIRST_READ);
- String offsetDiffBetweenFirstAndSecondRead = getMeanMetricValue(fileType, AVG_OFFSET_DIFF_BETWEEN_FIRST_AND_SECOND_READ);
-
- if (NON_PARQUET.equals(fileType)) {
- sizeReadByFirstRead += CHAR_UNDERSCORE + getMeanMetricValue(fileType, AVG_FIRST_OFFSET_DIFF);
- offsetDiffBetweenFirstAndSecondRead += CHAR_UNDERSCORE + getMeanMetricValue(fileType, AVG_SECOND_OFFSET_DIFF);
- }
-
- metricBuilder.append(CHAR_DOLLAR)
- .append(fileType)
- .append(FIRST_READ)
- .append(sizeReadByFirstRead)
- .append(SECOND_READ)
- .append(offsetDiffBetweenFirstAndSecondRead)
- .append(FILE_LENGTH)
- .append(getMeanMetricValue(fileType, AVG_FILE_LENGTH))
- .append(READ_LENGTH)
- .append(getMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED));
+ private String getOffsetDiffBetweenFirstAndSecondRead() {
+ return offsetDiffBetweenFirstAndSecondRead;
}
/**
- * Returns the read footer metrics for all file types.
+ * Returns the file type.
*
- * @return the read footer metrics as a string
+ * @return the file type
*/
- @Override
- public String toString() {
- StringBuilder readFooterMetric = new StringBuilder();
- appendMetrics(readFooterMetric, PARQUET);
- appendMetrics(readFooterMetric, NON_PARQUET);
- return readFooterMetric.toString();
+ private FileType getFileType() {
+ return fileType;
+ }
+ }
+
+ /**
+ * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ */
+ public AbfsReadFooterMetrics() {
+ IOStatisticsStore ioStatisticsStore = iostatisticsStore()
+ .withCounters(getMetricNames(TYPE_COUNTER))
+ .withMeanStatistics(getMetricNames(TYPE_MEAN))
+ .build();
+ setIOStatistics(ioStatisticsStore);
+ }
+
+ /**
+ * Constructor to initialize the IOStatisticsStore with counters and mean statistics,
+ * and a map to track file type metrics.
+ *
+ * @param fileTypeMetricsMap the map to track file type metrics
+ */
+ public AbfsReadFooterMetrics(Map<String, FileTypeMetrics> fileTypeMetricsMap) {
+ IOStatisticsStore ioStatisticsStore = iostatisticsStore()
+ .withCounters(getMetricNames(TYPE_COUNTER))
+ .withMeanStatistics(getMetricNames(TYPE_MEAN))
+ .build();
+ setIOStatistics(ioStatisticsStore);
+ this.fileTypeMetricsMap = fileTypeMetricsMap;
+ }
+
+ /**
+ * Returns the map of file type metrics.
+ *
+ * @return the map of file type metrics
+ */
+ public Map<String, FileTypeMetrics> getFileTypeMetricsMap() {
+ return fileTypeMetricsMap;
+ }
+
+ /**
+ * Returns the metric names for a specific statistic type.
+ *
+ * @param type the statistic type
+ * @return the metric names
+ */
+ private String[] getMetricNames(StatisticTypeEnum type) {
+ return Arrays.stream(AbfsReadFooterMetricsEnum.values())
+ .filter(
+ readFooterMetricsEnum -> readFooterMetricsEnum.getStatisticType()
+ .equals(type))
+ .flatMap(readFooterMetricsEnum ->
+ FILE.equals(readFooterMetricsEnum.getType())
+ ? FILE_TYPE_LIST.stream().map(fileType ->
+ getMetricName(fileType, readFooterMetricsEnum))
+ : Stream.of(readFooterMetricsEnum.getName()))
+ .toArray(String[]::new);
+ }
+
+ /**
+ * Returns the metric name for a specific file type and metric.
+ *
+ * @param fileType the type of the file
+ * @param readFooterMetricsEnum the metric to get the name for
+ * @return the metric name
+ */
+ private String getMetricName(FileType fileType,
+ AbfsReadFooterMetricsEnum readFooterMetricsEnum) {
+ if (fileType == null || readFooterMetricsEnum == null) {
+ LOG.error("File type or ABFS read footer metrics should not be null");
+ return EMPTY_STRING;
+ }
+ return fileType + COLON + readFooterMetricsEnum.getName();
+ }
+
+ /**
+ * Looks up the counter value for a specific metric.
+ *
+ * @param fileType the type of the file
+ * @param abfsReadFooterMetricsEnum the metric to look up
+ * @return the counter value
+ */
+ private long getCounterMetricValue(FileType fileType,
+ AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
+ return lookupCounterValue(
+ getMetricName(fileType, abfsReadFooterMetricsEnum));
+ }
+
+ /**
+ * Looks up the mean statistic value for a specific metric.
+ *
+ * @param fileType the type of the file
+ * @param abfsReadFooterMetricsEnum the metric to look up
+ * @return the mean statistic value
+ */
+ private String getMeanMetricValue(FileType fileType,
+ AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
+ return format(DOUBLE_PRECISION_FORMAT,
+ lookupMeanStatistic(
+ getMetricName(fileType, abfsReadFooterMetricsEnum)));
+ }
+
+ /**
+ * Increments the value of a specific metric.
+ *
+ * @param fileType the type of the file
+ * @param abfsReadFooterMetricsEnum the metric to increment
+ */
+ public void incrementMetricValue(FileType fileType,
+ AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
+ incCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ }
+
+ /**
+ * Adds a mean statistic value for a specific metric.
+ *
+ * @param fileType the type of the file
+ * @param abfsReadFooterMetricsEnum the metric to update
+ * @param value the new value of the metric
+ */
+ public void addMeanMetricValue(FileType fileType,
+ AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum,
+ long value) {
+ addMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum), value);
+ }
+
+ /**
+ * Returns the total number of files.
+ *
+ * @return the total number of files
+ */
+ public Long getTotalFiles() {
+ return getCounterMetricValue(PARQUET, TOTAL_FILES) + getCounterMetricValue(
+ NON_PARQUET, TOTAL_FILES);
+ }
+
+ /**
+ * Updates the map with a new file path identifier.
+ *
+ * @param filePathIdentifier the file path identifier
+ */
+ public void updateMap(String filePathIdentifier) {
+ fileTypeMetricsMap.computeIfAbsent(filePathIdentifier,
+ key -> new FileTypeMetrics());
+ }
+
+ /**
+ * Checks and updates the metrics for a given file read.
+ *
+ * @param filePathIdentifier the file path identifier
+ * @param len the length of the read
+ * @param contentLength the total content length of the file
+ * @param nextReadPos the position of the next read
+ */
+ public void updateReadMetrics(final String filePathIdentifier,
+ final int len,
+ final long contentLength,
+ final long nextReadPos) {
+ FileTypeMetrics fileTypeMetrics = fileTypeMetricsMap.computeIfAbsent(
+ filePathIdentifier, key -> new FileTypeMetrics());
+ if (fileTypeMetrics.getReadCount() == 0 || (
+ fileTypeMetrics.getReadCount() >= 1
+ && fileTypeMetrics.getCollectMetrics())) {
+ updateMetrics(fileTypeMetrics, len, contentLength, nextReadPos);
+ }
+ }
+
+ /**
+ * Updates metrics for a specific file identified by filePathIdentifier.
+ *
+ * @param fileTypeMetrics File metadata to know file type.
+ * @param len The length of the read operation.
+ * @param contentLength The total content length of the file.
+ * @param nextReadPos The position of the next read operation.
+ */
+ private void updateMetrics(FileTypeMetrics fileTypeMetrics,
+ int len,
+ long contentLength,
+ long nextReadPos) {
+ fileTypeMetrics.incrementReadCount();
+
+ long readCount = fileTypeMetrics.getReadCount();
+
+ if (readCount == 1) {
+ handleFirstRead(fileTypeMetrics, nextReadPos, len, contentLength);
+ } else if (readCount == 2) {
+ handleSecondRead(fileTypeMetrics, nextReadPos, len, contentLength);
+ } else {
+ handleFurtherRead(fileTypeMetrics, len);
+ }
+ }
+
+ /**
+ * Handles the first read operation by checking if the current read position is near the end of the file.
+ * If it is, updates the {@link FileTypeMetrics} object to enable metrics collection and records the first read's
+ * offset and size.
+ *
+ * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
+ * @param nextReadPos The position where the next read will start.
+ * @param len The length of the current read operation.
+ * @param contentLength The total length of the file content.
+ */
+ private void handleFirstRead(FileTypeMetrics fileTypeMetrics,
+ long nextReadPos,
+ int len,
+ long contentLength) {
+ if (nextReadPos
Review Comment:
Condition should be in same line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650737407
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -876,6 +894,308 @@ public void testIsNonEmptyDirectory() throws IOException {
false, 1, true);
}
+ /**
+ * Test to verify that in case metric account is not set,
+ * metric collection is enabled with default metric format
+ * and account url.
+ *
+ * @throws Exception in case of any failure
+ */
+ @Test
+ public void testMetricAccountFallback() throws Exception {
+ Configuration configuration = getRawConfiguration();
+ configuration.setBoolean(
+ AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION, false);
+ configuration.setBoolean(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, false);
+ configuration.unset(FS_AZURE_METRIC_ACCOUNT_NAME);
+ configuration.unset(FS_AZURE_METRIC_ACCOUNT_KEY);
+ configuration.unset(FS_AZURE_METRIC_FORMAT);
+ configuration.setBoolean(FS_AZURE_ALWAYS_USE_HTTPS, true);
+ final AzureBlobFileSystem fs = getFileSystem(configuration);
+ Assertions.assertThat(
+ fs.getAbfsStore().getAbfsConfiguration().getMetricFormat())
+ .describedAs(
+ "In case metric format is not set, metric format should "
+ + "be defaulted to internal metric format")
+ .isEqualTo(MetricFormat.INTERNAL_METRIC_FORMAT);
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().isMetricCollectionEnabled())
+ .describedAs(
+ "Metric collection should be enabled even if metric account is not set")
+ .isTrue();
+
+ Assertions.assertThat(
Review Comment:
If its INTERNAL_METRIC_FORMAT it should have both backoff and footer metrics right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2648034556
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ this.rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleAtFixedRate(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
+ }));
+ }
+
+ /**
+ * Get the singleton instance of AggregateMetricsManager.
+ *
+ * @param dispatchIntervalInMins Interval in minutes for dispatching metrics.
+ * @param permitsPerSecond Rate limit for dispatching metrics.
+ * @return Singleton instance of AggregateMetricsManager.
+ */
+ public static AggregateMetricsManager get(final long dispatchIntervalInMins,
+ final int permitsPerSecond) {
+ if (instance != null) {
+ return instance;
+ }
+
+ synchronized (AggregateMetricsManager.class) {
+ if (instance == null) {
+ try {
+ instance = new AggregateMetricsManager(
+ dispatchIntervalInMins, permitsPerSecond);
+ } catch (InvalidConfigurationValueException e) {
+ throw new RuntimeException(
+ "Failed to initialize AggregateMetricsManager", e);
+ }
+ }
+ return instance;
+ }
+ }
+
+ /**
+ * Register an AbfsClient with the manager.
+ * @param account Account name.
+ * @param abfsClient AbfsClient instance.
+ */
+ public void registerClient(String account, AbfsClient abfsClient) {
+ if (StringUtils.isEmpty(account) || abfsClient == null) {
+ return;
+ }
+
+ buckets.computeIfAbsent(account,
+ key -> new MetricsBucket(rateLimiter))
+ .registerClient(abfsClient);
+ }
+
+ /**
+ * Deregister an AbfsClient from the manager.
+ * @param account Account name.
+ * @param abfsClient AbfsClient instance.
+ * @return true if the client was deregistered, false otherwise.
+ */
+ public boolean deregisterClient(String account, AbfsClient abfsClient) {
+ if (StringUtils.isEmpty(account) || abfsClient == null) {
+ return false;
+ }
+
+ MetricsBucket bucket = buckets.get(account);
+ if (bucket == null) {
+ return false;
+ }
+
+ boolean isRemoved = bucket.deregisterClient(abfsClient);
+
+ if (bucket.isEmpty()) {
Review Comment:
There is a small race window between isEmpty() and remove(). Another thread may concurrently register a new client for the same account and reuse the bucket, but it can still be removed based on the earlier emptiness check. This makes the behavior timing-dependent and hard to reason about under concurrency.
You can use buckets.computeIfPresent() to perform the emptiness check and removal atomically, which avoids this race and keeps the map state consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2665169636
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1643,38 +1718,63 @@ private void resumeTimer() {
* Initiates a metric call to the Azure Blob FileSystem (ABFS) for retrieving file system properties.
* This method performs a HEAD request to the specified metric URL, using default headers and query parameters.
*
- * @param tracingContext The tracing context to be used for capturing tracing information.
+ * @param metricsData The tracing context to be used for capturing tracing information.
* @throws IOException throws IOException.
*/
- public void getMetricCall(TracingContext tracingContext) throws IOException {
- this.isSendMetricCall = true;
+ public void getMetricCall(String metricsData) throws IOException {
+ if (StringUtils.isEmpty(metricsData)) {
+ return;
+ }
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final AbfsUriQueryBuilder abfsUriQueryBuilder
+ = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM);
// Construct the URL for the metric call
// In case of blob storage, the URL is changed to DFS URL
- final URL url = UriUtils.changeUrlFromBlobToDfs(
- createRequestUrl(new URL(abfsMetricUrl),
- EMPTY_STRING, abfsUriQueryBuilder.toString()));
+ final URL url = createRequestUrl(getMetricsUrl(),
+ EMPTY_STRING, abfsUriQueryBuilder.toString());
final AbfsRestOperation op = getAbfsRestOperation(
- AbfsRestOperationType.GetFileSystemProperties,
- HTTP_METHOD_HEAD,
- url,
- requestHeaders);
+ AbfsRestOperationType.GetFileSystemProperties,
+ HTTP_METHOD_HEAD,
+ url,
+ requestHeaders);
+ TracingContext tracingContext = new TracingContext(
+ TracingContext.validateClientCorrelationID(
+ abfsConfiguration.getClientCorrelationId()),
+ fileSystemId, FSOperationType.GET_ATTR, true,
+ TracingHeaderFormat.AGGREGATED_METRICS_FORMAT,
+ null, metricsData);
try {
op.execute(tracingContext);
- } finally {
- this.isSendMetricCall = false;
+ } catch (AzureBlobFileSystemException e) {
+ // Ignore the exception and continue.
}
}
- public boolean isSendMetricCall() {
- return isSendMetricCall;
+ public boolean isMetricCollectionEnabled() {
+ return isMetricCollectionEnabled.get();
}
- public boolean isMetricCollectionEnabled() {
- return isMetricCollectionEnabled;
+ /**
+ * Getter for metric URL.
+ *
+ * @return metricUrl
+ */
+ @VisibleForTesting
+ public URL getMetricsUrl() {
+ return metricUrl;
+ }
+
+ /**
+ * Setter for metric URL.
+ * Converts blob URL to dfs URL in case of blob storage account.
+ *
+ * @param urlString to be set as metricUrl.
+ * @throws IOException if URL is malformed.
+ */
+ private void setMetricsUrl(String urlString) throws IOException {
+ metricUrl = UriUtils.changeUrlFromBlobToDfs(new URL(urlString));
Review Comment:
GetFileSystemProperties on root is only supported for DFS endpoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3674749764
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 9 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 45s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| -1 :x: | javadoc | 0m 45s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 39s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 21s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 13s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 36s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| -1 :x: | javadoc | 0m 31s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 20s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | shadedclient | 26m 24s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 19s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not generate ASF License warnings. |
| | | 105m 13s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Switch statement found in org.apache.hadoop.fs.azurebfs.AbfsCountersImpl.initializeMetrics(MetricFormat, AbfsConfiguration) where one case falls through to the next case At AbfsCountersImpl.java:AbfsConfiguration) where one case falls through to the next case At AbfsCountersImpl.java:[lines 212-214] |
| | org.apache.hadoop.fs.azurebfs.services.MetricsBucket.sendMetrics(AbfsClient, ConcurrentLinkedQueue) might ignore java.io.IOException At MetricsBucket.java: At MetricsBucket.java:[line 140] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 21d95b49fc9a 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ba2961c5f0111a0122da322e7c9f1da52b7cb2ee |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/testReport/ |
| Max. process+thread count | 633 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/5/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650666021
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
+ // This client is the last one — drain metrics now
+ batchToSend = metricsBuffer.getAndSet(new ConcurrentLinkedQueue<>());
+ isLastClient = true;
+ }
+
+ clients.remove(client);
+ }
+ if (isLastClient) {
+ sendMetrics(client, batchToSend);
+ }
+ return true;
+ }
+
+ /**
+ * Add a metric to the buffer.
+ * @param metric the metric to add
+ */
+ void addRequest(String metric) {
+ if (metric != null) {
+ metricsBuffer.get().add(metric);
+ }
+ }
+
+ /**
+ * Drain the metrics buffer and send if there are registered clients.
+ */
+ public void drainAndSendIfReady() {
+ AbfsClient client;
+
+ synchronized (this) {
+ if (clients.isEmpty()) {
+ return;
+ }
+ client = clients.iterator().next();
+ }
+
+ ConcurrentLinkedQueue<String> batch = metricsBuffer.getAndSet(
+ new ConcurrentLinkedQueue<>());
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ sendMetrics(client, batch);
+ }
+
+ // Send metrics outside synchronized block
+ private void sendMetrics(AbfsClient client,
+ ConcurrentLinkedQueue<String> batchToSend) {
+ // Send outside synchronized block
+ if (client != null && batchToSend != null && !batchToSend.isEmpty()) {
+ for (String chunk : splitListBySize(batchToSend, MAX_HEADER_SIZE)) {
+ rateLimiter.acquire(); // Rate limiting
Review Comment:
This can wait forever to acquire, should be time based ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2648083898
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ this.rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleAtFixedRate(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
+ }));
+ }
+
+ /**
+ * Get the singleton instance of AggregateMetricsManager.
+ *
+ * @param dispatchIntervalInMins Interval in minutes for dispatching metrics.
+ * @param permitsPerSecond Rate limit for dispatching metrics.
+ * @return Singleton instance of AggregateMetricsManager.
+ */
+ public static AggregateMetricsManager get(final long dispatchIntervalInMins,
Review Comment:
get method name is very generic, use better naming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2648081702
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ this.rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleAtFixedRate(
Review Comment:
Better to use scheduleWithFixedDelay as scheduleAtFixedRate can overlap executions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650644827
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
Review Comment:
The code checks whether the client is the last one before removing it, so another thread can change the client list in between, causing metrics to be sent at the wrong time or using a closing client.
Solution: first remove the client, then check if the client list is empty, and only then drain and send the remaining metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652389644
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -1643,38 +1718,63 @@ private void resumeTimer() {
* Initiates a metric call to the Azure Blob FileSystem (ABFS) for retrieving file system properties.
* This method performs a HEAD request to the specified metric URL, using default headers and query parameters.
*
- * @param tracingContext The tracing context to be used for capturing tracing information.
+ * @param metricsData The tracing context to be used for capturing tracing information.
* @throws IOException throws IOException.
*/
- public void getMetricCall(TracingContext tracingContext) throws IOException {
- this.isSendMetricCall = true;
+ public void getMetricCall(String metricsData) throws IOException {
+ if (StringUtils.isEmpty(metricsData)) {
+ return;
+ }
final List<AbfsHttpHeader> requestHeaders = createDefaultHeaders();
- final AbfsUriQueryBuilder abfsUriQueryBuilder = createDefaultUriQueryBuilder();
+ final AbfsUriQueryBuilder abfsUriQueryBuilder
+ = createDefaultUriQueryBuilder();
abfsUriQueryBuilder.addQuery(QUERY_PARAM_RESOURCE, FILESYSTEM);
// Construct the URL for the metric call
// In case of blob storage, the URL is changed to DFS URL
- final URL url = UriUtils.changeUrlFromBlobToDfs(
- createRequestUrl(new URL(abfsMetricUrl),
- EMPTY_STRING, abfsUriQueryBuilder.toString()));
+ final URL url = createRequestUrl(getMetricsUrl(),
+ EMPTY_STRING, abfsUriQueryBuilder.toString());
final AbfsRestOperation op = getAbfsRestOperation(
- AbfsRestOperationType.GetFileSystemProperties,
- HTTP_METHOD_HEAD,
- url,
- requestHeaders);
+ AbfsRestOperationType.GetFileSystemProperties,
+ HTTP_METHOD_HEAD,
+ url,
+ requestHeaders);
+ TracingContext tracingContext = new TracingContext(
+ TracingContext.validateClientCorrelationID(
+ abfsConfiguration.getClientCorrelationId()),
+ fileSystemId, FSOperationType.GET_ATTR, true,
+ TracingHeaderFormat.AGGREGATED_METRICS_FORMAT,
+ null, metricsData);
try {
op.execute(tracingContext);
- } finally {
- this.isSendMetricCall = false;
+ } catch (AzureBlobFileSystemException e) {
+ // Ignore the exception and continue.
}
}
- public boolean isSendMetricCall() {
- return isSendMetricCall;
+ public boolean isMetricCollectionEnabled() {
+ return isMetricCollectionEnabled.get();
}
- public boolean isMetricCollectionEnabled() {
- return isMetricCollectionEnabled;
+ /**
+ * Getter for metric URL.
+ *
+ * @return metricUrl
+ */
+ @VisibleForTesting
+ public URL getMetricsUrl() {
+ return metricUrl;
+ }
+
+ /**
+ * Setter for metric URL.
+ * Converts blob URL to dfs URL in case of blob storage account.
+ *
+ * @param urlString to be set as metricUrl.
+ * @throws IOException if URL is malformed.
+ */
+ private void setMetricsUrl(String urlString) throws IOException {
+ metricUrl = UriUtils.changeUrlFromBlobToDfs(new URL(urlString));
Review Comment:
why cant we send it for blob endpoints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3742473833
Java doc errors and Spot bugs are not from the changes in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2672107583
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,176 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleWithFixedDelay(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
+ }));
+ }
+
+ /**
+ * Get the singleton instance of AggregateMetricsManager.
+ *
+ * @param dispatchIntervalInMins Interval in minutes for dispatching metrics.
+ * @param permitsPerSecond Rate limit for dispatching metrics.
+ * @return Singleton instance of AggregateMetricsManager.
+ */
+ public static AggregateMetricsManager getInstance(final long dispatchIntervalInMins,
+ final int permitsPerSecond) {
+ if (instance != null) {
+ return instance;
+ }
+
+ synchronized (AggregateMetricsManager.class) {
+ if (instance == null) {
+ try {
+ instance = new AggregateMetricsManager(
+ dispatchIntervalInMins, permitsPerSecond);
+ } catch (InvalidConfigurationValueException e) {
+ throw new RuntimeException(
+ "Failed to initialize AggregateMetricsManager", e);
+ }
+ }
+ return instance;
+ }
+ }
+
+ /**
+ * Register an AbfsClient with the manager.
+ * @param account Account name.
+ * @param abfsClient AbfsClient instance.
+ */
+ public void registerClient(String account, AbfsClient abfsClient) {
+ if (StringUtils.isEmpty(account) || abfsClient == null) {
+ return;
+ }
+
+ buckets.computeIfAbsent(account,
+ key -> new MetricsBucket(rateLimiter))
+ .registerClient(abfsClient);
+ }
+
+ /**
+ * Deregister an AbfsClient from the manager.
+ * @param account Account name.
+ * @param abfsClient AbfsClient instance.
+ * @return true if the client was deregistered, false otherwise.
+ */
+ public boolean deregisterClient(String account, AbfsClient abfsClient) {
+ if (StringUtils.isEmpty(account) || abfsClient == null) {
+ return false;
+ }
+
+ AtomicBoolean isRemoved = new AtomicBoolean(false);
+
+ buckets.computeIfPresent(account, (key, bucket) -> {
+ // Deregister the client
+ isRemoved.set(bucket.deregisterClient(abfsClient));
+
+ // If bucket became empty, remove it atomically
+ return bucket.isEmpty() ? null : bucket;
+ });
+
+ return isRemoved.get();
+ }
+
+ /**
+ * Record metrics data for a specific account.
+ * @param accountName Account name.
+ * @param metricsData Metrics data to record.
+ */
+ public void recordMetric(String accountName, String metricsData) {
+ if (StringUtils.isEmpty(accountName)
Review Comment:
Should have null check for metricsData as we are returning null from metrics manager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3664082040
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 2m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 8 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 43s | | trunk passed |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| -1 :x: | javadoc | 0m 45s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 40s | [/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/branch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in trunk failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 24s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 28s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| -1 :x: | javadoc | 0m 31s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | spotbugs | 1m 22s | [/new-spotbugs-hadoop-tools_hadoop-azure.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/new-spotbugs-hadoop-tools_hadoop-azure.html) | hadoop-tools/hadoop-azure generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | shadedclient | 26m 14s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 18s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not generate ASF License warnings. |
| | | 110m 49s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-tools/hadoop-azure |
| | Switch statement found in org.apache.hadoop.fs.azurebfs.AbfsCountersImpl.initializeMetrics(MetricFormat) where one case falls through to the next case At AbfsCountersImpl.java:where one case falls through to the next case At AbfsCountersImpl.java:[lines 210-212] |
| | org.apache.hadoop.fs.azurebfs.services.MetricsBucket.sendMetrics(AbfsClient, ConcurrentLinkedQueue) might ignore java.io.IOException At MetricsBucket.java: At MetricsBucket.java:[line 140] |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 63ce14ddcd0b 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1af17c1a8ce5b3c1eef8f0504ab883674acfcbd9 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/testReport/ |
| Max. process+thread count | 637 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/3/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2642758374
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
+ public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
Review Comment:
Add jvadoc for all these with {@value} tag.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
+ public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
Review Comment:
Keep all the metric related configs name consistent with same prefix.
`fs.azure.metrics....`
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##########
@@ -136,6 +136,13 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_ENABLE_AUTOTHROTTLING = false;
public static final int DEFAULT_METRIC_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS = 60_000;
+ public static final boolean DEFAULT_METRICS_COLLECTION_ENABLED = true;
+ public static final boolean DEFAULT_SHOULD_EMIT_METRICS_ON_IDLE_TIME = false;
+ public static final long DEFAULT_METRICS_EMIT_THRESHOLD = 100_000L;
+ public static final long DEFAULT_METRICS_EMIT_THRESHOLD_INTERVAL_SECS = 60;
Review Comment:
Genrally names here also contains _FS_. Format was exact config variable name prefixed with DEFAULT_
I know this isn't followed everywhere but let's try to resolve this here.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -217,11 +233,13 @@ private AbfsClient(final URL baseUrl,
final AbfsConfiguration abfsConfiguration,
final EncryptionContextProvider encryptionContextProvider,
final AbfsClientContext abfsClientContext,
+ final String fileSystemId,
Review Comment:
I think its better to pass it as a part of client context similar to other client related fields.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
+ private static final String FOOTER_LENGTH = "20";
Review Comment:
Add comment as to why 20?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBackoffMetrics.java:
##########
@@ -75,18 +74,24 @@
* retry operations in Azure Blob File System (ABFS).
*/
public class AbfsBackoffMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsBackoffMetrics.class);
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsBackoffMetrics.class);
+
private static final List<RetryValue> RETRY_LIST = Arrays.asList(
- RetryValue.values());
+ RetryValue.values());
+
+ private final boolean isRetryMetricEnabled;
Review Comment:
What is the purpose of this?
Please add javadocs wherever necessary.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -281,42 +303,84 @@ private AbfsClient(final URL baseUrl,
new ThreadFactoryBuilder().setNameFormat("AbfsClient Lease Ops").setDaemon(true).build();
this.executorService = MoreExecutors.listeningDecorator(
HadoopExecutors.newScheduledThreadPool(this.abfsConfiguration.getNumLeaseThreads(), tf));
- this.metricFormat = abfsConfiguration.getMetricFormat();
+ this.isMetricCollectionEnabled.set(abfsConfiguration.isMetricsCollectionEnabled());
this.isMetricCollectionStopped = new AtomicBoolean(false);
this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
- if (StringUtils.isNotEmpty(metricFormat.toString())) {
- String metricAccountName = abfsConfiguration.getMetricAccount();
- String metricAccountKey = abfsConfiguration.getMetricAccountKey();
- if (StringUtils.isNotEmpty(metricAccountName) && StringUtils.isNotEmpty(metricAccountKey)) {
- isMetricCollectionEnabled = true;
- abfsCounters.initializeMetrics(metricFormat);
- int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- metricAccountName + " - account name is not fully qualified.");
+ if (isMetricCollectionEnabled()) {
Review Comment:
I feel like lot of metric related busuness logic is there in client constructor.
Let's create a separate MetricsManager class for AbfsCleint which will encapsulate all the timers and scedulers for metrics and exposes certain public methods for client to call like init, close etc as needed.
This way code will look much cleaner
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -281,42 +303,84 @@ private AbfsClient(final URL baseUrl,
new ThreadFactoryBuilder().setNameFormat("AbfsClient Lease Ops").setDaemon(true).build();
this.executorService = MoreExecutors.listeningDecorator(
HadoopExecutors.newScheduledThreadPool(this.abfsConfiguration.getNumLeaseThreads(), tf));
- this.metricFormat = abfsConfiguration.getMetricFormat();
+ this.isMetricCollectionEnabled.set(abfsConfiguration.isMetricsCollectionEnabled());
this.isMetricCollectionStopped = new AtomicBoolean(false);
this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
- if (StringUtils.isNotEmpty(metricFormat.toString())) {
- String metricAccountName = abfsConfiguration.getMetricAccount();
- String metricAccountKey = abfsConfiguration.getMetricAccountKey();
- if (StringUtils.isNotEmpty(metricAccountName) && StringUtils.isNotEmpty(metricAccountKey)) {
- isMetricCollectionEnabled = true;
- abfsCounters.initializeMetrics(metricFormat);
- int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- metricAccountName + " - account name is not fully qualified.");
+ if (isMetricCollectionEnabled()) {
+ try {
+ String metricAccountName = abfsConfiguration.getMetricAccount();
+ String metricAccountKey = abfsConfiguration.getMetricAccountKey();
+ this.metricFormat = abfsConfiguration.getMetricFormat();
+ abfsCounters.initializeMetrics(metricFormat, getAbfsConfiguration());
+ if (isNotEmpty(metricAccountName) && isNotEmpty(
+ metricAccountKey)) {
+ int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
Review Comment:
Add a test around this exception if not already there
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java:
##########
@@ -338,7 +339,7 @@ void completeExecute(TracingContext tracingContext)
LOG.debug("Rest operation {} failed with failureReason: {}. Retrying with retryCount = {}, retryPolicy: {} and sleepInterval: {}",
operationType, failureReason, retryCount, retryPolicy.getAbbreviation(), retryInterval);
if (abfsBackoffMetrics != null) {
- updateBackoffTimeMetrics(retryCount, sleepDuration);
+ updateBackoffTimeMetrics(retryCount, retryInterval);
Review Comment:
Why this change?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##########
@@ -136,6 +136,13 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_ENABLE_AUTOTHROTTLING = false;
public static final int DEFAULT_METRIC_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS = 60_000;
+ public static final boolean DEFAULT_METRICS_COLLECTION_ENABLED = true;
Review Comment:
We want to keep it enabled by default for all Cxs?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -281,42 +303,84 @@ private AbfsClient(final URL baseUrl,
new ThreadFactoryBuilder().setNameFormat("AbfsClient Lease Ops").setDaemon(true).build();
this.executorService = MoreExecutors.listeningDecorator(
HadoopExecutors.newScheduledThreadPool(this.abfsConfiguration.getNumLeaseThreads(), tf));
- this.metricFormat = abfsConfiguration.getMetricFormat();
+ this.isMetricCollectionEnabled.set(abfsConfiguration.isMetricsCollectionEnabled());
this.isMetricCollectionStopped = new AtomicBoolean(false);
this.metricAnalysisPeriod = abfsConfiguration.getMetricAnalysisTimeout();
this.metricIdlePeriod = abfsConfiguration.getMetricIdleTimeout();
- if (StringUtils.isNotEmpty(metricFormat.toString())) {
- String metricAccountName = abfsConfiguration.getMetricAccount();
- String metricAccountKey = abfsConfiguration.getMetricAccountKey();
- if (StringUtils.isNotEmpty(metricAccountName) && StringUtils.isNotEmpty(metricAccountKey)) {
- isMetricCollectionEnabled = true;
- abfsCounters.initializeMetrics(metricFormat);
- int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
- if (dotIndex <= 0) {
- throw new InvalidUriException(
- metricAccountName + " - account name is not fully qualified.");
+ if (isMetricCollectionEnabled()) {
+ try {
+ String metricAccountName = abfsConfiguration.getMetricAccount();
+ String metricAccountKey = abfsConfiguration.getMetricAccountKey();
+ this.metricFormat = abfsConfiguration.getMetricFormat();
+ abfsCounters.initializeMetrics(metricFormat, getAbfsConfiguration());
+ if (isNotEmpty(metricAccountName) && isNotEmpty(
+ metricAccountKey)) {
+ int dotIndex = metricAccountName.indexOf(AbfsHttpConstants.DOT);
+ if (dotIndex <= 0) {
+ throw new InvalidUriException(
+ metricAccountName + " - account name is not fully qualified.");
+ }
+ try {
+ metricSharedkeyCredentials = new SharedKeyCredentials(
+ metricAccountName.substring(0, dotIndex),
+ metricAccountKey);
+ hasSeparateMetricAccount = true;
+ setMetricsUrl(metricAccountName.startsWith(HTTPS_SCHEME)
+ ? metricAccountName : HTTPS_SCHEME + COLON
+ + FORWARD_SLASH + FORWARD_SLASH + metricAccountName);
+ } catch (IllegalArgumentException e) {
+ throw new IOException(
+ "Exception while initializing metric credentials ", e);
+ }
+ } else {
+ setMetricsUrl(baseUrlString.substring(0, indexLastForwardSlash + 1));
}
- try {
- metricSharedkeyCredentials = new SharedKeyCredentials(
- metricAccountName.substring(0, dotIndex),
- metricAccountKey);
- } catch (IllegalArgumentException e) {
- throw new IOException("Exception while initializing metric credentials ", e);
+
+ // register the client to Aggregated Metrics Manager
+ this.aggregateMetricsManager.registerClient(accountName, this);
+
+ // Metrics emitter scheduler
+ this.metricsEmitScheduler
+ = Executors.newSingleThreadScheduledExecutor();
+ // run every 1 minute to check the metrics count
+ this.metricsEmitScheduler.scheduleAtFixedRate(
Review Comment:
There are 2 separate schedulers being added here seems like.
Each client has its own scheduler and then the singleton metric manager class also has one?
Is this as per design?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
Review Comment:
Not needed anymore?
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -372,19 +436,26 @@ public AbfsClient(final URL baseUrl,
final SASTokenProvider sasTokenProvider,
final EncryptionContextProvider encryptionContextProvider,
final AbfsClientContext abfsClientContext,
- final AbfsServiceType abfsServiceType)
+ final String fileSystemId, final AbfsServiceType abfsServiceType)
throws IOException {
this(baseUrl, sharedKeyCredentials, abfsConfiguration,
- encryptionContextProvider, abfsClientContext, abfsServiceType);
+ encryptionContextProvider, abfsClientContext, fileSystemId, abfsServiceType);
this.sasTokenProvider = sasTokenProvider;
this.tokenProvider = tokenProvider;
}
@Override
public void close() throws IOException {
- if (isMetricCollectionEnabled && runningTimerTask != null) {
- runningTimerTask.cancel();
- timer.cancel();
+ if (isMetricCollectionEnabled()) {
Review Comment:
If not already done, verify that after FS close all the threads are properly getting shutdown and no leak is there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647911913
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBackoffMetrics.java:
##########
@@ -305,7 +328,9 @@ public String toString() {
return EMPTY_STRING;
}
StringBuilder metricBuilder = new StringBuilder();
- getRetryMetrics(metricBuilder);
+ if (isRetryMetricEnabled) {
Review Comment:
I made this change as per the discussion we had with the team. We can discuss this offline and if agreed by all I will revert the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2648018584
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
Review Comment:
rateLimiter is declared as static but is initialized in the constructor using permitsPerSecond.
This means the rate limiter ends up being global to the JVM, even though its value comes from instance-level configuration. In practice, whichever code initializes the manager first decides the rate, and any later calls to get() with different values are silently ignored.
This makes the behavior dependent on initialization order, which is hard to reason about and can lead to surprising behavior in tests or when multiple ABFS clients are created with different configs.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
Review Comment:
rateLimiter is declared as static but is initialized in the constructor using permitsPerSecond.
This means the rate limiter ends up being global to the JVM, even though its value comes from instance-level configuration. In practice, whichever code initializes the manager first decides the rate, and any later calls to get() with different values are silently ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652915933
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TracingHeaderFormat.java:
##########
@@ -23,6 +23,9 @@ public enum TracingHeaderFormat {
TWO_ID_FORMAT, // <correlation-id>:<client-req-id>
- ALL_ID_FORMAT; // <client-correlation-id>:<client-req-id>:<filesystem-id>
+ ALL_ID_FORMAT, // <client-correlation-id>:<client-req-id>:<filesystem-id>
// :<primary-req-id>:<stream-id>:<hdfs-operation>:<retry-count>
+
+ AGGREGATED_METRICS_FORMAT; // <client-correlation-id>:<filesystem-id>
Review Comment:
nit: we can have the comment in same line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650576514
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
+ private static final String FOOTER_LENGTH = "20";
+
+ private static final List<FileType> FILE_TYPE_LIST =
+ Arrays.asList(FileType.values());
+
+ private Map<String, FileTypeMetrics> fileTypeMetricsMap =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Inner class to handle file type checks.
+ */
+ private static final class FileTypeMetrics {
+
+ private final AtomicBoolean collectMetrics;
+
+ private final AtomicBoolean collectMetricsForNextRead;
+
+ private final AtomicBoolean collectLenMetrics;
+
+ private final AtomicLong readCount;
+
+ private final AtomicLong offsetOfFirstRead;
+
+ private FileType fileType = null;
+
+ private String sizeReadByFirstRead;
+
+ private String offsetDiffBetweenFirstAndSecondRead;
/**
- * Inner class to handle file type checks.
+ * Constructor to initialize the file type metrics.
*/
- private static final class FileTypeMetrics {
- private final AtomicBoolean collectMetrics;
- private final AtomicBoolean collectMetricsForNextRead;
- private final AtomicBoolean collectLenMetrics;
- private final AtomicLong readCount;
- private final AtomicLong offsetOfFirstRead;
- private FileType fileType = null;
- private String sizeReadByFirstRead;
- private String offsetDiffBetweenFirstAndSecondRead;
-
- /**
- * Constructor to initialize the file type metrics.
- */
- private FileTypeMetrics() {
- collectMetrics = new AtomicBoolean(false);
- collectMetricsForNextRead = new AtomicBoolean(false);
- collectLenMetrics = new AtomicBoolean(false);
- readCount = new AtomicLong(0);
- offsetOfFirstRead = new AtomicLong(0);
- }
-
- /**
- * Updates the file type based on the metrics collected.
- */
- private void updateFileType() {
- if (fileType == null) {
- fileType = collectMetrics.get() && readCount.get() >= 2
- && haveEqualValues(sizeReadByFirstRead)
- && haveEqualValues(offsetDiffBetweenFirstAndSecondRead) ? PARQUET : NON_PARQUET;
- }
- }
-
- /**
- * Checks if the given value has equal parts.
- *
- * @param value the value to check
- * @return true if the value has equal parts, false otherwise
- */
- private boolean haveEqualValues(String value) {
- String[] parts = value.split("_");
- return parts.length == 2
- && parts[0].equals(parts[1]);
- }
-
- /**
- * Increments the read count.
- */
- private void incrementReadCount() {
- readCount.incrementAndGet();
- }
-
- /**
- * Returns the read count.
- *
- * @return the read count
- */
- private long getReadCount() {
- return readCount.get();
- }
-
- /**
- * Sets the collect metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetrics(boolean collect) {
- collectMetrics.set(collect);
- }
-
- /**
- * Returns the collect metrics flag.
- *
- * @return the collect metrics flag
- */
- private boolean getCollectMetrics() {
- return collectMetrics.get();
- }
-
- /**
- * Sets the collect metrics for the next read flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetricsForNextRead(boolean collect) {
- collectMetricsForNextRead.set(collect);
- }
-
- /**
- * Returns the collect metrics for the next read flag.
- *
- * @return the collect metrics for the next read flag
- */
- private boolean getCollectMetricsForNextRead() {
- return collectMetricsForNextRead.get();
- }
-
- /**
- * Returns the collect length metrics flag.
- *
- * @return the collect length metrics flag
- */
- private boolean getCollectLenMetrics() {
- return collectLenMetrics.get();
- }
-
- /**
- * Sets the collect length metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectLenMetrics(boolean collect) {
- collectLenMetrics.set(collect);
- }
-
- /**
- * Sets the offset of the first read.
- *
- * @param offset the value to set
- */
- private void setOffsetOfFirstRead(long offset) {
- offsetOfFirstRead.set(offset);
- }
-
- /**
- * Returns the offset of the first read.
- *
- * @return the offset of the first read
- */
- private long getOffsetOfFirstRead() {
- return offsetOfFirstRead.get();
- }
-
- /**
- * Sets the size read by the first read.
- *
- * @param size the value to set
- */
- private void setSizeReadByFirstRead(String size) {
- sizeReadByFirstRead = size;
- }
-
- /**
- * Returns the size read by the first read.
- *
- * @return the size read by the first read
- */
- private String getSizeReadByFirstRead() {
- return sizeReadByFirstRead;
- }
-
- /**
- * Sets the offset difference between the first and second read.
- *
- * @param offsetDiff the value to set
- */
- private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
- offsetDiffBetweenFirstAndSecondRead = offsetDiff;
- }
-
- /**
- * Returns the offset difference between the first and second read.
- *
- * @return the offset difference between the first and second read
- */
- private String getOffsetDiffBetweenFirstAndSecondRead() {
- return offsetDiffBetweenFirstAndSecondRead;
- }
-
- /**
- * Returns the file type.
- *
- * @return the file type
- */
- private FileType getFileType() {
- return fileType;
- }
+ private FileTypeMetrics() {
+ collectMetrics = new AtomicBoolean(false);
+ collectMetricsForNextRead = new AtomicBoolean(false);
+ collectLenMetrics = new AtomicBoolean(false);
+ readCount = new AtomicLong(0);
+ offsetOfFirstRead = new AtomicLong(0);
}
/**
- * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ * Updates the file type based on the metrics collected.
*/
- public AbfsReadFooterMetrics() {
- IOStatisticsStore ioStatisticsStore = iostatisticsStore()
- .withCounters(getMetricNames(TYPE_COUNTER))
- .withMeanStatistics(getMetricNames(TYPE_MEAN))
- .build();
- setIOStatistics(ioStatisticsStore);
+ private void updateFileType() {
+ if (fileType == null) {
+ fileType = collectMetrics.get() && readCount.get() >= 2
+ && haveEqualValues(sizeReadByFirstRead)
+ && haveEqualValues(offsetDiffBetweenFirstAndSecondRead)
+ ? PARQUET
+ : NON_PARQUET;
+ }
}
/**
- * Returns the metric names for a specific statistic type.
+ * Checks if the given value has equal parts.
*
- * @param type the statistic type
- * @return the metric names
+ * @param value the value to check
+ * @return true if the value has equal parts, false otherwise
*/
- private String[] getMetricNames(StatisticTypeEnum type) {
- return Arrays.stream(AbfsReadFooterMetricsEnum.values())
- .filter(readFooterMetricsEnum -> readFooterMetricsEnum.getStatisticType().equals(type))
- .flatMap(readFooterMetricsEnum ->
- FILE.equals(readFooterMetricsEnum.getType())
- ? FILE_TYPE_LIST.stream().map(fileType ->
- getMetricName(fileType, readFooterMetricsEnum))
- : Stream.of(readFooterMetricsEnum.getName()))
- .toArray(String[]::new);
+ private boolean haveEqualValues(String value) {
+ String[] parts = value.split("_");
Review Comment:
Add constant for underscore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652213545
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
Review Comment:
we can do sm like
abfsReadFooterMetrics = new AbfsReadFooterMetrics(
abfsReadFooterMetrics == null ? null : abfsReadFooterMetrics.getFileTypeMetricsMap()
);
we would have a single constructor for AbfsReadFooterMetrics then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3737432494
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 21s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/10/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 28s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 51s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| -1 :x: | javadoc | 0m 32s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/10/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/10/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | spotbugs | 1m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 21s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 21s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not generate ASF License warnings. |
| | | 105m 32s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8dcfc3b0bbc7 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2320a9c5e289fd5a751201eb5ef74cf776d0af02 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/10/testReport/ |
| Max. process+thread count | 612 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/10/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#issuecomment-3727506198
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 15m 24s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 10 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 1m 23s | [/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/9/artifact/out/branch-spotbugs-hadoop-tools_hadoop-azure-warnings.html) | hadoop-tools/hadoop-azure in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 26m 30s | | branch has no errors when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 52s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| -1 :x: | javadoc | 0m 32s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/9/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 30s | [/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/9/artifact/out/patch-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt) | hadoop-azure in the patch failed with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 20s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 3m 21s | | hadoop-azure in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not generate ASF License warnings. |
| | | 120m 37s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/8137 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e23ba3e596d0 5.15.0-161-generic #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 655dcb1d03ea24095a9cbb5ea9f62a0a8b6ffe40 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions | /usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/9/testReport/ |
| Max. process+thread count | 631 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8137/9/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 merged PR #8137:
URL: https://github.com/apache/hadoop/pull/8137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650719598
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
+ // This client is the last one — drain metrics now
+ batchToSend = metricsBuffer.getAndSet(new ConcurrentLinkedQueue<>());
+ isLastClient = true;
+ }
+
+ clients.remove(client);
+ }
+ if (isLastClient) {
+ sendMetrics(client, batchToSend);
+ }
+ return true;
+ }
+
+ /**
+ * Add a metric to the buffer.
+ * @param metric the metric to add
+ */
+ void addRequest(String metric) {
+ if (metric != null) {
+ metricsBuffer.get().add(metric);
+ }
+ }
+
+ /**
+ * Drain the metrics buffer and send if there are registered clients.
+ */
+ public void drainAndSendIfReady() {
+ AbfsClient client;
+
+ synchronized (this) {
+ if (clients.isEmpty()) {
+ return;
+ }
+ client = clients.iterator().next();
+ }
+
+ ConcurrentLinkedQueue<String> batch = metricsBuffer.getAndSet(
+ new ConcurrentLinkedQueue<>());
+ if (batch.isEmpty()) {
+ return;
+ }
+
+ sendMetrics(client, batch);
+ }
+
+ // Send metrics outside synchronized block
+ private void sendMetrics(AbfsClient client,
+ ConcurrentLinkedQueue<String> batchToSend) {
+ // Send outside synchronized block
+ if (client != null && batchToSend != null && !batchToSend.isEmpty()) {
+ for (String chunk : splitListBySize(batchToSend, MAX_HEADER_SIZE)) {
+ rateLimiter.acquire(); // Rate limiting
+ try {
+ client.getMetricCall(chunk);
+ } catch (IOException ignored) {}
+ }
+ }
+ }
+
+ // Check if there are no registered clients
+ public boolean isEmpty() {
+ return clients.isEmpty();
+ }
+
+ /**
+ * Split the list of metrics into chunks that fit within maxChars.
+ * Each metric is wrapped in square brackets and separated by colons.
+ */
+ private static List<String> splitListBySize(
+ ConcurrentLinkedQueue<String> items, long maxChars) {
+
+ if (items.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ List<String> result = new ArrayList<>();
+ StringBuilder sb = new StringBuilder();
+
+ for (String s : new ArrayList<>(items)) {
+ String wrapped = "[" + s + "]";
Review Comment:
use constants for colon and brackets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652904931
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/SimpleRateLimiter.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+
+public final class SimpleRateLimiter {
+
+ // Interval between permits in nanoseconds.
+ private final long intervalNanos;
+
+ // Next allowed time to acquire a permit in nanoseconds.
+ private long nextAllowedTime;
Review Comment:
should we have this as atomic/volatile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650734028
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/SimpleRateLimiter.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+
+public final class SimpleRateLimiter {
+
+ // Interval between permits in nanoseconds.
+ private final long intervalNanos;
+
+ // Next allowed time to acquire a permit in nanoseconds.
+ private long nextAllowedTime;
+
+ /** Number of nanoseconds in one second. */
+ private static final long NANOS_PER_SECOND = 1_000_000_000L;
+
+ /**
+ * Constructs a SimpleRateLimiter that allows the specified number of
+ * permits per second.
+ *
+ * @param permitsPerSecond Number of permits allowed per second.
+ * @throws InvalidConfigurationValueException if permitsPerSecond is
+ * less than or equal to zero.
+ */
+ public SimpleRateLimiter(int permitsPerSecond)
+ throws InvalidConfigurationValueException {
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+ this.intervalNanos = NANOS_PER_SECOND / permitsPerSecond;
+ this.nextAllowedTime = System.nanoTime();
+ }
+
+ /**
+ * Acquires a permit from the rate limiter, blocking until one is available.
+ */
+ public synchronized void acquire() {
Review Comment:
This method can block forever because it waits in an infinite loop while holding a synchronized lock, which can stall other threads and delay shutdown.
It also handles interrupts weakly and offers no timeout or non-blocking option, making it unsafe for critical paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650771888
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAggregateMetricsManager.java:
##########
@@ -0,0 +1,710 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+
+public class TestAggregateMetricsManager extends AbstractAbfsIntegrationTest {
+
+ /** Number of nanoseconds in one millisecond. */
+ private static final long NANOS_PER_MILLISECOND = 1_000_000L;
+
+ // The manager under test
+ private final AggregateMetricsManager manager;
+
+ /**
+ * Constructor for TestAggregateMetricsManager.
+ *
+ * @throws Exception if an error occurs during setup
+ */
+ public TestAggregateMetricsManager() throws Exception {
+ super();
+ manager = AggregateMetricsManager.get(10, 3);
+ }
+
+ /**
+ * Creates a fully instrumented TracingContext useful for metric dispatch tests.
+ */
+ private String getMetricsData() {
+ return UUID.randomUUID() + COLON + UUID.randomUUID() + COLON + "#BO:";
+ }
+
+ /**
+ * Wraps an AbfsClient instance in a spy and counts metric call invocations.
+ */
+ private AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,
+ AtomicInteger counter)
+ throws IOException {
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ counter.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+
+ return client;
+ }
+
+ /**
+ * Verifies that multiple recordMetric calls result in exactly one aggregated
+ * dispatch within a 1-second dispatch window.
+ */
+ @Test
+ public void testRecordMetric() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 1 aggregated metrics send")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Verifies that aggregated metrics for the same account are sent
+ * once per dispatch window, meaning 2 windows → 2 sends.
+ */
+ @Test
+ public void testRecordMetricTwoWindows() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+
+ manager.deregisterClient("acc1", client);
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 1 aggregated sends")
+ .isEqualTo(1);
+
+ manager.registerClient("acc1", client);
+ // Second window
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 2 aggregated sends")
+ .isEqualTo(2);
+ }
+
+ /**
+ * Ensures that recordMetric handles invalid input without exceptions.
+ */
+ @Test
+ public void testRecordMetricWithNulls() throws Exception {
+ manager.recordMetric(null, null);
+ manager.recordMetric("", null);
+ manager.recordMetric("acc", null);
+ manager.recordMetric("acc", null);
+ }
+
+ /**
+ * Ensures that metrics for separate accounts still respect global
+ * rate limiting but send independently within the same window.
+ */
+ @Test
+ public void testMultipleAccounts() throws Exception {
+
+ AtomicInteger calls1 = new AtomicInteger();
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger();
+ AbfsClient client2 = spyClient(Mockito.spy(
+ (AzureBlobFileSystem) AzureBlobFileSystem.newInstance(
+ getRawConfiguration())), calls2);
+
+ manager.registerClient("acc1", client1);
+ manager.registerClient("acc2", client2);
+ manager.recordMetric("acc1", getMetricsData());
+ manager.recordMetric("acc2", getMetricsData());
+ manager.deregisterClient("acc1", client1);
+ manager.deregisterClient("acc2", client2);
+
+ Assertions.assertThat(calls1.get())
+ .describedAs("Account 1 dispatched once")
+ .isEqualTo(1);
+
+ Assertions.assertThat(calls2.get())
+ .describedAs("Account 2 dispatched once")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Tests concurrent registration, metric recording, and deregistration
+ * of multiple clients for the same account.
+ */
+ @Test
+ public void testMultipleClientsRegistryInParallel() throws Exception {
+ AtomicInteger calls1 = new AtomicInteger(0);
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger(0);
+ AbfsClient client2 = spyClient(Mockito.spy(this.getFileSystem()), calls2);
+
+ CountDownLatch latch = new CountDownLatch(5);
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ for (int i = 0; i < 10; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ //wait for all threads to finish
+ latch.await();
+
+ Assertions.assertThat((calls1.get() == 1) ^ (calls2.get() == 1))
+ .describedAs("Exactly one client should send metrics")
+ .isTrue();
+ }
+
+ /**
+ * Tests deregistering a nonexistent client.
+ */
+ @Test
+ public void testDeregisterNonexistentClient() throws IOException {
+ AbfsClient client = Mockito.spy(
+ this.getFileSystem().getAbfsStore().getClient());
+ // Should not throw
+ boolean isRemoved = manager.deregisterClient("nonexistentAccount", client);
+ Assertions.assertThat(isRemoved)
+ .describedAs("Deregistering nonexistent client should return false")
+ .isFalse();
+ }
+
+ /**
+ * Tests that when the aggregated metric data exceeds the buffer size,
+ * multiple dispatches occur as expected.
+ */
+ @Test
+ public void testMultipleMetricCallsInCaseDataIsMoreThanBufferSize()
+ throws Exception {
+ final int metricsDataSize1
+ = 927; // size of aggregated data for first 3 calls
+ final int metricsDataSize2 = 115; // size of aggregated data for last call
+ final int numberOfMetrics = 25; // total metrics to send
+ AtomicInteger calls = new AtomicInteger(0);
+ AzureBlobFileSystem azureBlobFileSystem = Mockito.spy(this.getFileSystem());
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ String data = (String) inv.getArguments()[0];
+ if (calls.get() < 3) { // first three calls, data size will be 927 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 927 chars")
+ .isEqualTo(metricsDataSize1);
+ } else { // last call, data size will be 115 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 115 chars")
+ .isEqualTo(metricsDataSize2);
+ }
+ calls.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < numberOfMetrics; i++) {
+ manager.recordMetric("acc1", getMetricsData()
+ + "$OT=163$RT=6.024%$TRNR=2543$TR=2706"); // each data is 113 chars
+ }
+ manager.deregisterClient("acc1", client);
+
+ // 113 + 2 ([,]) = 115 chars per metric, 115 * 25 = 2875 chars total + 24 (:) = 2899 chars
+ // 1st -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 2nd -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 3rd -> 115 * 9 = 920 chars + 7 (:) = 927 chars
+ // 4th -> remaining
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 3 aggregated metrics send")
+ .isEqualTo(4);
+ }
+
+ /**
+ * Verifies that when multiple clients send metrics concurrently,
+ * the global rate limiter enforces spacing between dispatches.
+ */
+ @Test
+ public void testRateLimitMetricCalls()
+ throws IOException, InterruptedException {
+ final int permitsPerSecond = 3;
+ final long minIntervalMs = 1_000 / permitsPerSecond; // 333ms
+ final double toleranceMs = 15; // allow 15ms jitter
+ final int numClients = 10;
+
+ // Store timestamps for each client
+ final List<AtomicLong> times = new ArrayList<>();
+ AbfsClient[] abfsClients = new AbfsClient[numClients];
+
+ for (int i = 0; i < numClients; i++) {
+ AtomicLong time = new AtomicLong();
+ times.add(time);
+
+ AbfsClient client = createSpiedClient(time);
+ abfsClients[i] = client;
+ manager.registerClient("acc" + i, client);
+ }
+
+ // Record metrics for all clients
+ for (int i = 0; i < numClients; i++) {
+ manager.recordMetric("acc" + i, getMetricsData());
+ }
+
+ // Deregister all clients concurrently
+ CountDownLatch latch = new CountDownLatch(numClients);
+ for (int i = 0; i < numClients; i++) {
+ final int idx = i;
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc" + idx,
+ abfsClients[idx]); // pass time for demonstration if needed
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+ }
+ latch.await();
+
+ // Check that interval between any two timestamps is ≥ minIntervalMs - tolerance
+ for (int i = 0; i < times.size(); i++) {
+ for (int j = i + 1; j < times.size(); j++) {
+ double diffMs = Math.abs(times.get(i).get() - times.get(j).get())
+ / (double) NANOS_PER_MILLISECOND;
+ Assertions.assertThat(diffMs)
+ .describedAs(
+ "Expected at least %d ms (tolerance %.3f) between metric sends",
+ minIntervalMs, toleranceMs)
+ .isGreaterThanOrEqualTo(minIntervalMs - toleranceMs);
+ }
+ }
+ }
+
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit0()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 1 (kept exactly as you asked)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " AzureBlobFileSystem fs = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AbfsClient client = spyClient(Mockito.spy(fs), calls1);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client);\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " System.exit(0);\n"
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,\n"
+ + " AtomicInteger counter) throws IOException {\n"
+ + "\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + "\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, true, 0);
+ }
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit after multiple
+ * clients and deregistrations.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit1()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 2 (final, multi-FS version)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AtomicInteger calls2 = new AtomicInteger();\n"
+ + " AtomicInteger calls3 = new AtomicInteger();\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " AzureBlobFileSystem fs1 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs2 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs3 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + "\n"
+ + " AbfsClient client1 = spyClient(Mockito.spy(fs1), calls1);\n"
+ + " AbfsClient client2 = spyClient(Mockito.spy(fs2), calls2);\n"
+ + " AbfsClient client3 = spyClient(Mockito.spy(fs3), calls3);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client1);\n"
+ + " mgr.registerClient(\"acc1\", client2);\n"
+ + " mgr.registerClient(\"acc1\", client3);\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m3\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m4\");\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m5\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m6\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " mgr.deregisterClient(\"acc1\", client3);\n"
+ + " mgr.deregisterClient(\"acc1\", client2);\n"
+ + " mgr.deregisterClient(\"acc1\", client1);\n"
+ + " System.out.println(\"BEFORE_EXIT1\");\n"
+ + " System.exit(1);\n"
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,\n"
+ + " AtomicInteger counter) throws IOException {\n"
+ + "\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + "\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, true, 1);
+ }
+
+ /**
+ * Tests that the shutdown hook does not flush metrics on JVM crash.
+ */
+ @Test
+ void testAggregatedMetricsManagerWithJVMCrash() throws Exception {
+ final int crashExitCode = 134;
+ // -------------------------------
+ // Program 3 (JVM Crash)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import java.lang.reflect.*;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + "\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " // Track how many times metrics flush\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AtomicInteger calls2 = new AtomicInteger();\n"
+ + " AtomicInteger calls3 = new AtomicInteger();\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " // Initialize 3 separate file system instances\n"
+ + " AzureBlobFileSystem fs1 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs2 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs3 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + "\n"
+ + " // Create 3 spy clients\n"
+ + " AbfsClient client1 = spyClient(Mockito.spy(fs1), calls1);\n"
+ + " AbfsClient client2 = spyClient(Mockito.spy(fs2), calls2);\n"
+ + " AbfsClient client3 = spyClient(Mockito.spy(fs3), calls3);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client1);\n"
+ + " mgr.registerClient(\"acc2\", client2);\n"
+ + " mgr.registerClient(\"acc3\", client3);\n"
+ + "\n"
+ + " // Produce metrics on all clients\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + " mgr.recordMetric(\"acc2\", \"m3\");\n"
+ + " mgr.recordMetric(\"acc2\", \"m4\");\n"
+ + " mgr.recordMetric(\"acc3\", \"m5\");\n"
+ + " mgr.recordMetric(\"acc3\", \"m6\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " crashJvm();\n"
+ + " }\n"
+ + "\n"
+ + " private static void crashJvm() throws Exception {\n"
+ + " Field f = sun.misc.Unsafe.class.getDeclaredField(\"theUnsafe\");\n"
+ + " f.setAccessible(true);\n"
+ + " sun.misc.Unsafe unsafe = (sun.misc.Unsafe) f.get(null);\n"
+ + " unsafe.putAddress(0, 0); // SIGSEGV → Immediate JVM crash\n"
+ // 128 + 6 = 134 (exitcode for SIGABRT)
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem, AtomicInteger counter) throws IOException {\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, false, crashExitCode);
+ }
+
+ /**
+ * Compiles and runs a Java program in a separate JVM, capturing its output.
+ *
+ * @param program The Java program source code as a string.
+ * @param expectMetricsFlush Whether to expect metrics flush output.
+ * @throws IOException If an I/O error occurs.
+ * @throws InterruptedException If the thread is interrupted while waiting.
+ */
+ private void runProgramAndCaptureOutput(String program,
+ boolean expectMetricsFlush, int expectedExitCode)
+ throws IOException, InterruptedException {
+ Path tempFile = Files.createTempFile("ShutdownTestProg", ".java");
+ Files.write(tempFile, program.getBytes(StandardCharsets.UTF_8));
+
+ Path javaFile = tempFile.getParent().resolve("ShutdownTestProg.java");
+ Files.move(tempFile, javaFile, StandardCopyOption.REPLACE_EXISTING);
+
+ Process javac = new ProcessBuilder(
+ "javac",
+ "-classpath", System.getProperty("java.class.path"),
+ javaFile.toAbsolutePath().toString())
+ .redirectErrorStream(true)
+ .start();
+
+ String compileOutput = readProcessOutput(javac);
+ javac.waitFor();
Review Comment:
Add timed wait like
if (!javac.waitFor(30, TimeUnit.SECONDS)) {
javac.destroyForcibly();
throw new AssertionError("javac timed out");
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2650772349
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAggregateMetricsManager.java:
##########
@@ -0,0 +1,710 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;
+import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;
+
+import static org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.COLON;
+
+public class TestAggregateMetricsManager extends AbstractAbfsIntegrationTest {
+
+ /** Number of nanoseconds in one millisecond. */
+ private static final long NANOS_PER_MILLISECOND = 1_000_000L;
+
+ // The manager under test
+ private final AggregateMetricsManager manager;
+
+ /**
+ * Constructor for TestAggregateMetricsManager.
+ *
+ * @throws Exception if an error occurs during setup
+ */
+ public TestAggregateMetricsManager() throws Exception {
+ super();
+ manager = AggregateMetricsManager.get(10, 3);
+ }
+
+ /**
+ * Creates a fully instrumented TracingContext useful for metric dispatch tests.
+ */
+ private String getMetricsData() {
+ return UUID.randomUUID() + COLON + UUID.randomUUID() + COLON + "#BO:";
+ }
+
+ /**
+ * Wraps an AbfsClient instance in a spy and counts metric call invocations.
+ */
+ private AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,
+ AtomicInteger counter)
+ throws IOException {
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ counter.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+
+ return client;
+ }
+
+ /**
+ * Verifies that multiple recordMetric calls result in exactly one aggregated
+ * dispatch within a 1-second dispatch window.
+ */
+ @Test
+ public void testRecordMetric() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 1 aggregated metrics send")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Verifies that aggregated metrics for the same account are sent
+ * once per dispatch window, meaning 2 windows → 2 sends.
+ */
+ @Test
+ public void testRecordMetricTwoWindows() throws Exception {
+ AtomicInteger calls = new AtomicInteger(0);
+ AbfsClient client = spyClient(Mockito.spy(this.getFileSystem()), calls);
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+
+ manager.deregisterClient("acc1", client);
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 1 aggregated sends")
+ .isEqualTo(1);
+
+ manager.registerClient("acc1", client);
+ // Second window
+ for (int i = 0; i < 5; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ manager.deregisterClient("acc1", client);
+
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected 2 aggregated sends")
+ .isEqualTo(2);
+ }
+
+ /**
+ * Ensures that recordMetric handles invalid input without exceptions.
+ */
+ @Test
+ public void testRecordMetricWithNulls() throws Exception {
+ manager.recordMetric(null, null);
+ manager.recordMetric("", null);
+ manager.recordMetric("acc", null);
+ manager.recordMetric("acc", null);
+ }
+
+ /**
+ * Ensures that metrics for separate accounts still respect global
+ * rate limiting but send independently within the same window.
+ */
+ @Test
+ public void testMultipleAccounts() throws Exception {
+
+ AtomicInteger calls1 = new AtomicInteger();
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger();
+ AbfsClient client2 = spyClient(Mockito.spy(
+ (AzureBlobFileSystem) AzureBlobFileSystem.newInstance(
+ getRawConfiguration())), calls2);
+
+ manager.registerClient("acc1", client1);
+ manager.registerClient("acc2", client2);
+ manager.recordMetric("acc1", getMetricsData());
+ manager.recordMetric("acc2", getMetricsData());
+ manager.deregisterClient("acc1", client1);
+ manager.deregisterClient("acc2", client2);
+
+ Assertions.assertThat(calls1.get())
+ .describedAs("Account 1 dispatched once")
+ .isEqualTo(1);
+
+ Assertions.assertThat(calls2.get())
+ .describedAs("Account 2 dispatched once")
+ .isEqualTo(1);
+ }
+
+ /**
+ * Tests concurrent registration, metric recording, and deregistration
+ * of multiple clients for the same account.
+ */
+ @Test
+ public void testMultipleClientsRegistryInParallel() throws Exception {
+ AtomicInteger calls1 = new AtomicInteger(0);
+ AbfsClient client1 = spyClient(Mockito.spy(this.getFileSystem()), calls1);
+
+ AtomicInteger calls2 = new AtomicInteger(0);
+ AbfsClient client2 = spyClient(Mockito.spy(this.getFileSystem()), calls2);
+
+ CountDownLatch latch = new CountDownLatch(5);
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.registerClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ for (int i = 0; i < 10; i++) {
+ manager.recordMetric("acc1", getMetricsData());
+ }
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client1);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc1", client2);
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+
+ //wait for all threads to finish
+ latch.await();
+
+ Assertions.assertThat((calls1.get() == 1) ^ (calls2.get() == 1))
+ .describedAs("Exactly one client should send metrics")
+ .isTrue();
+ }
+
+ /**
+ * Tests deregistering a nonexistent client.
+ */
+ @Test
+ public void testDeregisterNonexistentClient() throws IOException {
+ AbfsClient client = Mockito.spy(
+ this.getFileSystem().getAbfsStore().getClient());
+ // Should not throw
+ boolean isRemoved = manager.deregisterClient("nonexistentAccount", client);
+ Assertions.assertThat(isRemoved)
+ .describedAs("Deregistering nonexistent client should return false")
+ .isFalse();
+ }
+
+ /**
+ * Tests that when the aggregated metric data exceeds the buffer size,
+ * multiple dispatches occur as expected.
+ */
+ @Test
+ public void testMultipleMetricCallsInCaseDataIsMoreThanBufferSize()
+ throws Exception {
+ final int metricsDataSize1
+ = 927; // size of aggregated data for first 3 calls
+ final int metricsDataSize2 = 115; // size of aggregated data for last call
+ final int numberOfMetrics = 25; // total metrics to send
+ AtomicInteger calls = new AtomicInteger(0);
+ AzureBlobFileSystem azureBlobFileSystem = Mockito.spy(this.getFileSystem());
+ AzureBlobFileSystemStore store = Mockito.spy(
+ azureBlobFileSystem.getAbfsStore());
+ Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();
+ AbfsClient client = Mockito.spy(store.getClient());
+ Mockito.doReturn(client).when(store).getClient();
+
+ Mockito.doAnswer(inv -> {
+ String data = (String) inv.getArguments()[0];
+ if (calls.get() < 3) { // first three calls, data size will be 927 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 927 chars")
+ .isEqualTo(metricsDataSize1);
+ } else { // last call, data size will be 115 chars
+ Assertions.assertThat(data.length())
+ .describedAs("Aggregated metric data size should be 115 chars")
+ .isEqualTo(metricsDataSize2);
+ }
+ calls.incrementAndGet();
+ return null;
+ }).when(client).getMetricCall(Mockito.any());
+ manager.registerClient("acc1", client);
+ for (int i = 0; i < numberOfMetrics; i++) {
+ manager.recordMetric("acc1", getMetricsData()
+ + "$OT=163$RT=6.024%$TRNR=2543$TR=2706"); // each data is 113 chars
+ }
+ manager.deregisterClient("acc1", client);
+
+ // 113 + 2 ([,]) = 115 chars per metric, 115 * 25 = 2875 chars total + 24 (:) = 2899 chars
+ // 1st -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 2nd -> 115 * 8 = 920 chars + 7 (:) = 927 chars
+ // 3rd -> 115 * 9 = 920 chars + 7 (:) = 927 chars
+ // 4th -> remaining
+ Assertions.assertThat(calls.get())
+ .describedAs("Expected exactly 3 aggregated metrics send")
+ .isEqualTo(4);
+ }
+
+ /**
+ * Verifies that when multiple clients send metrics concurrently,
+ * the global rate limiter enforces spacing between dispatches.
+ */
+ @Test
+ public void testRateLimitMetricCalls()
+ throws IOException, InterruptedException {
+ final int permitsPerSecond = 3;
+ final long minIntervalMs = 1_000 / permitsPerSecond; // 333ms
+ final double toleranceMs = 15; // allow 15ms jitter
+ final int numClients = 10;
+
+ // Store timestamps for each client
+ final List<AtomicLong> times = new ArrayList<>();
+ AbfsClient[] abfsClients = new AbfsClient[numClients];
+
+ for (int i = 0; i < numClients; i++) {
+ AtomicLong time = new AtomicLong();
+ times.add(time);
+
+ AbfsClient client = createSpiedClient(time);
+ abfsClients[i] = client;
+ manager.registerClient("acc" + i, client);
+ }
+
+ // Record metrics for all clients
+ for (int i = 0; i < numClients; i++) {
+ manager.recordMetric("acc" + i, getMetricsData());
+ }
+
+ // Deregister all clients concurrently
+ CountDownLatch latch = new CountDownLatch(numClients);
+ for (int i = 0; i < numClients; i++) {
+ final int idx = i;
+ new Thread(() -> {
+ try {
+ manager.deregisterClient("acc" + idx,
+ abfsClients[idx]); // pass time for demonstration if needed
+ } finally {
+ latch.countDown();
+ }
+ }).start();
+ }
+ latch.await();
+
+ // Check that interval between any two timestamps is ≥ minIntervalMs - tolerance
+ for (int i = 0; i < times.size(); i++) {
+ for (int j = i + 1; j < times.size(); j++) {
+ double diffMs = Math.abs(times.get(i).get() - times.get(j).get())
+ / (double) NANOS_PER_MILLISECOND;
+ Assertions.assertThat(diffMs)
+ .describedAs(
+ "Expected at least %d ms (tolerance %.3f) between metric sends",
+ minIntervalMs, toleranceMs)
+ .isGreaterThanOrEqualTo(minIntervalMs - toleranceMs);
+ }
+ }
+ }
+
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit0()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 1 (kept exactly as you asked)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " AzureBlobFileSystem fs = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AbfsClient client = spyClient(Mockito.spy(fs), calls1);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client);\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " System.exit(0);\n"
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,\n"
+ + " AtomicInteger counter) throws IOException {\n"
+ + "\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + "\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, true, 0);
+ }
+
+ /**
+ * Tests that the shutdown hook flushes metrics on JVM exit after multiple
+ * clients and deregistrations.
+ */
+ @Test
+ public void testAggregatedMetricsManagerWithJVMExit1()
+ throws IOException, InterruptedException {
+ // -------------------------------
+ // Program 2 (final, multi-FS version)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AtomicInteger calls2 = new AtomicInteger();\n"
+ + " AtomicInteger calls3 = new AtomicInteger();\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " AzureBlobFileSystem fs1 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs2 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs3 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + "\n"
+ + " AbfsClient client1 = spyClient(Mockito.spy(fs1), calls1);\n"
+ + " AbfsClient client2 = spyClient(Mockito.spy(fs2), calls2);\n"
+ + " AbfsClient client3 = spyClient(Mockito.spy(fs3), calls3);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client1);\n"
+ + " mgr.registerClient(\"acc1\", client2);\n"
+ + " mgr.registerClient(\"acc1\", client3);\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m3\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m4\");\n"
+ + "\n"
+ + " mgr.recordMetric(\"acc1\", \"m5\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m6\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " mgr.deregisterClient(\"acc1\", client3);\n"
+ + " mgr.deregisterClient(\"acc1\", client2);\n"
+ + " mgr.deregisterClient(\"acc1\", client1);\n"
+ + " System.out.println(\"BEFORE_EXIT1\");\n"
+ + " System.exit(1);\n"
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem,\n"
+ + " AtomicInteger counter) throws IOException {\n"
+ + "\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + "\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, true, 1);
+ }
+
+ /**
+ * Tests that the shutdown hook does not flush metrics on JVM crash.
+ */
+ @Test
+ void testAggregatedMetricsManagerWithJVMCrash() throws Exception {
+ final int crashExitCode = 134;
+ // -------------------------------
+ // Program 3 (JVM Crash)
+ // -------------------------------
+ String program =
+ "import org.apache.hadoop.fs.azurebfs.services.*;\n"
+ + "import org.apache.hadoop.fs.azurebfs.utils.*;\n"
+ + "import org.apache.hadoop.conf.Configuration;\n"
+ + "import org.apache.hadoop.fs.FileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem;\n"
+ + "import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore;\n"
+ + "import java.util.*;\n"
+ + "import java.util.concurrent.atomic.AtomicInteger;\n"
+ + "import java.io.IOException;\n"
+ + "import java.net.URI;\n"
+ + "import java.lang.reflect.*;\n"
+ + "import org.mockito.Mockito;\n"
+ + "\n"
+ + "public class ShutdownTestProg {\n"
+ + "\n"
+ + " public static void main(String[] args) throws Exception {\n"
+ + " AggregateMetricsManager mgr = AggregateMetricsManager.get(10, 3);\n"
+ + "\n"
+ + " // Track how many times metrics flush\n"
+ + " AtomicInteger calls1 = new AtomicInteger();\n"
+ + " AtomicInteger calls2 = new AtomicInteger();\n"
+ + " AtomicInteger calls3 = new AtomicInteger();\n"
+ + "\n"
+ + " URI uri = new URI(\"abfss://test@manishtestfnsnew.dfs.core.windows.net\");\n"
+ + " Configuration config = new Configuration();\n"
+ + "\n"
+ + " // Initialize 3 separate file system instances\n"
+ + " AzureBlobFileSystem fs1 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs2 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + " AzureBlobFileSystem fs3 = (AzureBlobFileSystem) FileSystem.newInstance(uri, config);\n"
+ + "\n"
+ + " // Create 3 spy clients\n"
+ + " AbfsClient client1 = spyClient(Mockito.spy(fs1), calls1);\n"
+ + " AbfsClient client2 = spyClient(Mockito.spy(fs2), calls2);\n"
+ + " AbfsClient client3 = spyClient(Mockito.spy(fs3), calls3);\n"
+ + "\n"
+ + " mgr.registerClient(\"acc1\", client1);\n"
+ + " mgr.registerClient(\"acc2\", client2);\n"
+ + " mgr.registerClient(\"acc3\", client3);\n"
+ + "\n"
+ + " // Produce metrics on all clients\n"
+ + " mgr.recordMetric(\"acc1\", \"m1\");\n"
+ + " mgr.recordMetric(\"acc1\", \"m2\");\n"
+ + " mgr.recordMetric(\"acc2\", \"m3\");\n"
+ + " mgr.recordMetric(\"acc2\", \"m4\");\n"
+ + " mgr.recordMetric(\"acc3\", \"m5\");\n"
+ + " mgr.recordMetric(\"acc3\", \"m6\");\n"
+ + "\n"
+ + " System.out.println(\"BEFORE_EXIT\");\n"
+ + " crashJvm();\n"
+ + " }\n"
+ + "\n"
+ + " private static void crashJvm() throws Exception {\n"
+ + " Field f = sun.misc.Unsafe.class.getDeclaredField(\"theUnsafe\");\n"
+ + " f.setAccessible(true);\n"
+ + " sun.misc.Unsafe unsafe = (sun.misc.Unsafe) f.get(null);\n"
+ + " unsafe.putAddress(0, 0); // SIGSEGV → Immediate JVM crash\n"
+ // 128 + 6 = 134 (exitcode for SIGABRT)
+ + " }\n"
+ + "\n"
+ + " private static AbfsClient spyClient(AzureBlobFileSystem azureBlobFileSystem, AtomicInteger counter) throws IOException {\n"
+ + " AzureBlobFileSystemStore store = Mockito.spy(azureBlobFileSystem.getAbfsStore());\n"
+ + " Mockito.doReturn(store).when(azureBlobFileSystem).getAbfsStore();\n"
+ + " AbfsClient client = Mockito.spy(store.getClient());\n"
+ + " Mockito.doReturn(client).when(store).getClient();\n"
+ + "\n"
+ + " Mockito.doAnswer(inv -> {\n"
+ + " counter.incrementAndGet();\n"
+ + " System.out.println(\"FLUSH:\" + inv.getArguments()[0]);\n"
+ + " return null;\n"
+ + " }).when(client).getMetricCall(Mockito.any());\n"
+ + "\n"
+ + " return client;\n"
+ + " }\n"
+ + "}\n";
+
+ runProgramAndCaptureOutput(program, false, crashExitCode);
+ }
+
+ /**
+ * Compiles and runs a Java program in a separate JVM, capturing its output.
+ *
+ * @param program The Java program source code as a string.
+ * @param expectMetricsFlush Whether to expect metrics flush output.
+ * @throws IOException If an I/O error occurs.
+ * @throws InterruptedException If the thread is interrupted while waiting.
+ */
+ private void runProgramAndCaptureOutput(String program,
+ boolean expectMetricsFlush, int expectedExitCode)
+ throws IOException, InterruptedException {
+ Path tempFile = Files.createTempFile("ShutdownTestProg", ".java");
+ Files.write(tempFile, program.getBytes(StandardCharsets.UTF_8));
+
+ Path javaFile = tempFile.getParent().resolve("ShutdownTestProg.java");
+ Files.move(tempFile, javaFile, StandardCopyOption.REPLACE_EXISTING);
+
+ Process javac = new ProcessBuilder(
+ "javac",
+ "-classpath", System.getProperty("java.class.path"),
+ javaFile.toAbsolutePath().toString())
+ .redirectErrorStream(true)
+ .start();
+
+ String compileOutput = readProcessOutput(javac);
+ javac.waitFor();
+
+ Assertions.assertThat(javac.exitValue())
+ .withFailMessage("Compilation failed:\n" + compileOutput)
+ .isEqualTo(0);
+
+ String classpath = javaFile.getParent().toAbsolutePath()
+ + File.pathSeparator
+ + System.getProperty("java.class.path");
+
+ Process javaProc = new ProcessBuilder("java",
+ "-XX:ErrorFile=/tmp/no_hs_err_%p.log",
+ "-classpath", classpath,
+ "ShutdownTestProg")
+ .redirectErrorStream(true)
+ .start();
+
+ String output = readProcessOutput(javaProc);
+ int exitCode = javaProc.waitFor();
Review Comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "manika137 (via GitHub)" <gi...@apache.org>.
manika137 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2652550484
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
+ private static final String FOOTER_LENGTH = "20";
+
+ private static final List<FileType> FILE_TYPE_LIST =
+ Arrays.asList(FileType.values());
+
+ private Map<String, FileTypeMetrics> fileTypeMetricsMap =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Inner class to handle file type checks.
+ */
+ private static final class FileTypeMetrics {
+
+ private final AtomicBoolean collectMetrics;
+
+ private final AtomicBoolean collectMetricsForNextRead;
+
+ private final AtomicBoolean collectLenMetrics;
+
+ private final AtomicLong readCount;
+
+ private final AtomicLong offsetOfFirstRead;
+
+ private FileType fileType = null;
+
+ private String sizeReadByFirstRead;
+
+ private String offsetDiffBetweenFirstAndSecondRead;
/**
- * Inner class to handle file type checks.
+ * Constructor to initialize the file type metrics.
*/
- private static final class FileTypeMetrics {
- private final AtomicBoolean collectMetrics;
- private final AtomicBoolean collectMetricsForNextRead;
- private final AtomicBoolean collectLenMetrics;
- private final AtomicLong readCount;
- private final AtomicLong offsetOfFirstRead;
- private FileType fileType = null;
- private String sizeReadByFirstRead;
- private String offsetDiffBetweenFirstAndSecondRead;
-
- /**
- * Constructor to initialize the file type metrics.
- */
- private FileTypeMetrics() {
- collectMetrics = new AtomicBoolean(false);
- collectMetricsForNextRead = new AtomicBoolean(false);
- collectLenMetrics = new AtomicBoolean(false);
- readCount = new AtomicLong(0);
- offsetOfFirstRead = new AtomicLong(0);
- }
-
- /**
- * Updates the file type based on the metrics collected.
- */
- private void updateFileType() {
- if (fileType == null) {
- fileType = collectMetrics.get() && readCount.get() >= 2
- && haveEqualValues(sizeReadByFirstRead)
- && haveEqualValues(offsetDiffBetweenFirstAndSecondRead) ? PARQUET : NON_PARQUET;
- }
- }
-
- /**
- * Checks if the given value has equal parts.
- *
- * @param value the value to check
- * @return true if the value has equal parts, false otherwise
- */
- private boolean haveEqualValues(String value) {
- String[] parts = value.split("_");
- return parts.length == 2
- && parts[0].equals(parts[1]);
- }
-
- /**
- * Increments the read count.
- */
- private void incrementReadCount() {
- readCount.incrementAndGet();
- }
-
- /**
- * Returns the read count.
- *
- * @return the read count
- */
- private long getReadCount() {
- return readCount.get();
- }
-
- /**
- * Sets the collect metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetrics(boolean collect) {
- collectMetrics.set(collect);
- }
-
- /**
- * Returns the collect metrics flag.
- *
- * @return the collect metrics flag
- */
- private boolean getCollectMetrics() {
- return collectMetrics.get();
- }
-
- /**
- * Sets the collect metrics for the next read flag.
- *
- * @param collect the value to set
- */
- private void setCollectMetricsForNextRead(boolean collect) {
- collectMetricsForNextRead.set(collect);
- }
-
- /**
- * Returns the collect metrics for the next read flag.
- *
- * @return the collect metrics for the next read flag
- */
- private boolean getCollectMetricsForNextRead() {
- return collectMetricsForNextRead.get();
- }
-
- /**
- * Returns the collect length metrics flag.
- *
- * @return the collect length metrics flag
- */
- private boolean getCollectLenMetrics() {
- return collectLenMetrics.get();
- }
-
- /**
- * Sets the collect length metrics flag.
- *
- * @param collect the value to set
- */
- private void setCollectLenMetrics(boolean collect) {
- collectLenMetrics.set(collect);
- }
-
- /**
- * Sets the offset of the first read.
- *
- * @param offset the value to set
- */
- private void setOffsetOfFirstRead(long offset) {
- offsetOfFirstRead.set(offset);
- }
-
- /**
- * Returns the offset of the first read.
- *
- * @return the offset of the first read
- */
- private long getOffsetOfFirstRead() {
- return offsetOfFirstRead.get();
- }
-
- /**
- * Sets the size read by the first read.
- *
- * @param size the value to set
- */
- private void setSizeReadByFirstRead(String size) {
- sizeReadByFirstRead = size;
- }
-
- /**
- * Returns the size read by the first read.
- *
- * @return the size read by the first read
- */
- private String getSizeReadByFirstRead() {
- return sizeReadByFirstRead;
- }
-
- /**
- * Sets the offset difference between the first and second read.
- *
- * @param offsetDiff the value to set
- */
- private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
- offsetDiffBetweenFirstAndSecondRead = offsetDiff;
- }
-
- /**
- * Returns the offset difference between the first and second read.
- *
- * @return the offset difference between the first and second read
- */
- private String getOffsetDiffBetweenFirstAndSecondRead() {
- return offsetDiffBetweenFirstAndSecondRead;
- }
-
- /**
- * Returns the file type.
- *
- * @return the file type
- */
- private FileType getFileType() {
- return fileType;
- }
+ private FileTypeMetrics() {
+ collectMetrics = new AtomicBoolean(false);
+ collectMetricsForNextRead = new AtomicBoolean(false);
+ collectLenMetrics = new AtomicBoolean(false);
+ readCount = new AtomicLong(0);
+ offsetOfFirstRead = new AtomicLong(0);
}
/**
- * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ * Updates the file type based on the metrics collected.
*/
- public AbfsReadFooterMetrics() {
- IOStatisticsStore ioStatisticsStore = iostatisticsStore()
- .withCounters(getMetricNames(TYPE_COUNTER))
- .withMeanStatistics(getMetricNames(TYPE_MEAN))
- .build();
- setIOStatistics(ioStatisticsStore);
+ private void updateFileType() {
+ if (fileType == null) {
+ fileType = collectMetrics.get() && readCount.get() >= 2
+ && haveEqualValues(sizeReadByFirstRead)
+ && haveEqualValues(offsetDiffBetweenFirstAndSecondRead)
+ ? PARQUET
+ : NON_PARQUET;
+ }
}
/**
- * Returns the metric names for a specific statistic type.
+ * Checks if the given value has equal parts.
*
- * @param type the statistic type
- * @return the metric names
+ * @param value the value to check
+ * @return true if the value has equal parts, false otherwise
*/
- private String[] getMetricNames(StatisticTypeEnum type) {
- return Arrays.stream(AbfsReadFooterMetricsEnum.values())
- .filter(readFooterMetricsEnum -> readFooterMetricsEnum.getStatisticType().equals(type))
- .flatMap(readFooterMetricsEnum ->
- FILE.equals(readFooterMetricsEnum.getType())
- ? FILE_TYPE_LIST.stream().map(fileType ->
- getMetricName(fileType, readFooterMetricsEnum))
- : Stream.of(readFooterMetricsEnum.getName()))
- .toArray(String[]::new);
+ private boolean haveEqualValues(String value) {
+ String[] parts = value.split("_");
+ return parts.length == 2
+ && parts[0].equals(parts[1]);
}
/**
- * Returns the metric name for a specific file type and metric.
- *
- * @param fileType the type of the file
- * @param readFooterMetricsEnum the metric to get the name for
- * @return the metric name
+ * Increments the read count.
*/
- private String getMetricName(FileType fileType,
- AbfsReadFooterMetricsEnum readFooterMetricsEnum) {
- if (fileType == null || readFooterMetricsEnum == null) {
- LOG.error("File type or ABFS read footer metrics should not be null");
- return EMPTY_STRING;
- }
- return fileType + COLON + readFooterMetricsEnum.getName();
+ private void incrementReadCount() {
+ readCount.incrementAndGet();
}
/**
- * Looks up the counter value for a specific metric.
+ * Returns the read count.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to look up
- * @return the counter value
+ * @return the read count
*/
- private long getCounterMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- return lookupCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ private long getReadCount() {
+ return readCount.get();
}
/**
- * Looks up the mean statistic value for a specific metric.
+ * Sets the collect metrics flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to look up
- * @return the mean statistic value
+ * @param collect the value to set
*/
- private String getMeanMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- return format(DOUBLE_PRECISION_FORMAT,
- lookupMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum)));
+ private void setCollectMetrics(boolean collect) {
+ collectMetrics.set(collect);
}
/**
- * Increments the value of a specific metric.
+ * Returns the collect metrics flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to increment
+ * @return the collect metrics flag
*/
- public void incrementMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum) {
- incCounterValue(getMetricName(fileType, abfsReadFooterMetricsEnum));
+ private boolean getCollectMetrics() {
+ return collectMetrics.get();
}
/**
- * Adds a mean statistic value for a specific metric.
+ * Sets the collect metrics for the next read flag.
*
- * @param fileType the type of the file
- * @param abfsReadFooterMetricsEnum the metric to update
- * @param value the new value of the metric
+ * @param collect the value to set
*/
- public void addMeanMetricValue(FileType fileType,
- AbfsReadFooterMetricsEnum abfsReadFooterMetricsEnum,
- long value) {
- addMeanStatistic(getMetricName(fileType, abfsReadFooterMetricsEnum), value);
+ private void setCollectMetricsForNextRead(boolean collect) {
+ collectMetricsForNextRead.set(collect);
}
/**
- * Returns the total number of files.
+ * Returns the collect metrics for the next read flag.
*
- * @return the total number of files
+ * @return the collect metrics for the next read flag
*/
- public Long getTotalFiles() {
- return getCounterMetricValue(PARQUET, TOTAL_FILES) + getCounterMetricValue(NON_PARQUET, TOTAL_FILES);
+ private boolean getCollectMetricsForNextRead() {
+ return collectMetricsForNextRead.get();
}
/**
- * Updates the map with a new file path identifier.
+ * Returns the collect length metrics flag.
*
- * @param filePathIdentifier the file path identifier
+ * @return the collect length metrics flag
*/
- public void updateMap(String filePathIdentifier) {
- fileTypeMetricsMap.computeIfAbsent(filePathIdentifier, key -> new FileTypeMetrics());
+ private boolean getCollectLenMetrics() {
+ return collectLenMetrics.get();
}
/**
- * Checks and updates the metrics for a given file read.
+ * Sets the collect length metrics flag.
*
- * @param filePathIdentifier the file path identifier
- * @param len the length of the read
- * @param contentLength the total content length of the file
- * @param nextReadPos the position of the next read
+ * @param collect the value to set
*/
- public void updateReadMetrics(final String filePathIdentifier,
- final int len,
- final long contentLength,
- final long nextReadPos) {
- FileTypeMetrics fileTypeMetrics = fileTypeMetricsMap.computeIfAbsent(filePathIdentifier, key -> new FileTypeMetrics());
- if (fileTypeMetrics.getReadCount() == 0 || (fileTypeMetrics.getReadCount() >= 1 && fileTypeMetrics.getCollectMetrics())) {
- updateMetrics(fileTypeMetrics, len, contentLength, nextReadPos);
- }
+ private void setCollectLenMetrics(boolean collect) {
+ collectLenMetrics.set(collect);
}
/**
- * Updates metrics for a specific file identified by filePathIdentifier.
+ * Sets the offset of the first read.
*
- * @param fileTypeMetrics File metadata to know file type.
- * @param len The length of the read operation.
- * @param contentLength The total content length of the file.
- * @param nextReadPos The position of the next read operation.
+ * @param offset the value to set
*/
- private void updateMetrics(FileTypeMetrics fileTypeMetrics,
- int len,
- long contentLength,
- long nextReadPos) {
- fileTypeMetrics.incrementReadCount();
-
- long readCount = fileTypeMetrics.getReadCount();
-
- if (readCount == 1) {
- handleFirstRead(fileTypeMetrics, nextReadPos, len, contentLength);
- } else if (readCount == 2) {
- handleSecondRead(fileTypeMetrics, nextReadPos, len, contentLength);
- } else {
- handleFurtherRead(fileTypeMetrics, len);
- }
+ private void setOffsetOfFirstRead(long offset) {
+ offsetOfFirstRead.set(offset);
}
/**
- * Handles the first read operation by checking if the current read position is near the end of the file.
- * If it is, updates the {@link FileTypeMetrics} object to enable metrics collection and records the first read's
- * offset and size.
+ * Returns the offset of the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
- * @param nextReadPos The position where the next read will start.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @return the offset of the first read
*/
- private void handleFirstRead(FileTypeMetrics fileTypeMetrics,
- long nextReadPos,
- int len,
- long contentLength) {
- if (nextReadPos >= contentLength - (long) Integer.parseInt(FOOTER_LENGTH) * ONE_KB) {
- fileTypeMetrics.setCollectMetrics(true);
- fileTypeMetrics.setCollectMetricsForNextRead(true);
- fileTypeMetrics.setOffsetOfFirstRead(nextReadPos);
- fileTypeMetrics.setSizeReadByFirstRead(len + "_" + Math.abs(contentLength - nextReadPos));
- }
+ private long getOffsetOfFirstRead() {
+ return offsetOfFirstRead.get();
}
/**
- * Handles the second read operation by checking if metrics collection is enabled for the next read.
- * If it is, calculates the offset difference between the first and second reads, updates the {@link FileTypeMetrics}
- * object with this information, and sets the file type. Then, updates the metrics data.
+ * Sets the size read by the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object to update with metrics and read details.
- * @param nextReadPos The position where the next read will start.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @param size the value to set
*/
- private void handleSecondRead(FileTypeMetrics fileTypeMetrics,
- long nextReadPos,
- int len,
- long contentLength) {
- if (fileTypeMetrics.getCollectMetricsForNextRead()) {
- long offsetDiff = Math.abs(nextReadPos - fileTypeMetrics.getOffsetOfFirstRead());
- fileTypeMetrics.setOffsetDiffBetweenFirstAndSecondRead(len + "_" + offsetDiff);
- fileTypeMetrics.setCollectLenMetrics(true);
- fileTypeMetrics.updateFileType();
- updateMetricsData(fileTypeMetrics, len, contentLength);
- }
+ private void setSizeReadByFirstRead(String size) {
+ sizeReadByFirstRead = size;
}
/**
- * Handles further read operations beyond the second read. If metrics collection is enabled and the file type is set,
- * updates the read length requested and increments the read count for the specific file type.
+ * Returns the size read by the first read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object containing metrics and read details.
- * @param len The length of the current read operation.
+ * @return the size read by the first read
*/
- private void handleFurtherRead(FileTypeMetrics fileTypeMetrics, int len) {
- if (fileTypeMetrics.getCollectLenMetrics() && fileTypeMetrics.getFileType() != null) {
- FileType fileType = fileTypeMetrics.getFileType();
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, len);
- }
+ private String getSizeReadByFirstRead() {
+ return sizeReadByFirstRead;
}
/**
- * Updates the metrics data for a specific file identified by the {@link FileTypeMetrics} object.
- * This method calculates and updates various metrics such as read length requested, file length,
- * size read by the first read, and offset differences between reads.
+ * Sets the offset difference between the first and second read.
*
- * @param fileTypeMetrics The {@link FileTypeMetrics} object containing metrics and read details.
- * @param len The length of the current read operation.
- * @param contentLength The total length of the file content.
+ * @param offsetDiff the value to set
*/
- private void updateMetricsData(FileTypeMetrics fileTypeMetrics,
- int len,
- long contentLength) {
- long sizeReadByFirstRead = Long.parseLong(fileTypeMetrics.getSizeReadByFirstRead().split("_")[0]);
- long firstOffsetDiff = Long.parseLong(fileTypeMetrics.getSizeReadByFirstRead().split("_")[1]);
- long secondOffsetDiff = Long.parseLong(fileTypeMetrics.getOffsetDiffBetweenFirstAndSecondRead().split("_")[1]);
- FileType fileType = fileTypeMetrics.getFileType();
-
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, len);
- addMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED, sizeReadByFirstRead);
- addMeanMetricValue(fileType, AVG_FILE_LENGTH, contentLength);
- addMeanMetricValue(fileType, AVG_SIZE_READ_BY_FIRST_READ, sizeReadByFirstRead);
- addMeanMetricValue(fileType, AVG_OFFSET_DIFF_BETWEEN_FIRST_AND_SECOND_READ, len);
- addMeanMetricValue(fileType, AVG_FIRST_OFFSET_DIFF, firstOffsetDiff);
- addMeanMetricValue(fileType, AVG_SECOND_OFFSET_DIFF, secondOffsetDiff);
- incrementMetricValue(fileType, TOTAL_FILES);
+ private void setOffsetDiffBetweenFirstAndSecondRead(String offsetDiff) {
+ offsetDiffBetweenFirstAndSecondRead = offsetDiff;
}
/**
- * Appends the metrics for a specific file type to the given metric builder.
+ * Returns the offset difference between the first and second read.
*
- * @param metricBuilder the metric builder to append the metrics to
- * @param fileType the file type to append the metrics for
+ * @return the offset difference between the first and second read
*/
- private void appendMetrics(StringBuilder metricBuilder, FileType fileType) {
- long totalFiles = getCounterMetricValue(fileType, TOTAL_FILES);
- if (totalFiles <= 0) {
- return;
- }
-
- String sizeReadByFirstRead = getMeanMetricValue(fileType, AVG_SIZE_READ_BY_FIRST_READ);
- String offsetDiffBetweenFirstAndSecondRead = getMeanMetricValue(fileType, AVG_OFFSET_DIFF_BETWEEN_FIRST_AND_SECOND_READ);
-
- if (NON_PARQUET.equals(fileType)) {
- sizeReadByFirstRead += CHAR_UNDERSCORE + getMeanMetricValue(fileType, AVG_FIRST_OFFSET_DIFF);
- offsetDiffBetweenFirstAndSecondRead += CHAR_UNDERSCORE + getMeanMetricValue(fileType, AVG_SECOND_OFFSET_DIFF);
- }
-
- metricBuilder.append(CHAR_DOLLAR)
- .append(fileType)
- .append(FIRST_READ)
- .append(sizeReadByFirstRead)
- .append(SECOND_READ)
- .append(offsetDiffBetweenFirstAndSecondRead)
- .append(FILE_LENGTH)
- .append(getMeanMetricValue(fileType, AVG_FILE_LENGTH))
- .append(READ_LENGTH)
- .append(getMeanMetricValue(fileType, AVG_READ_LEN_REQUESTED));
+ private String getOffsetDiffBetweenFirstAndSecondRead() {
+ return offsetDiffBetweenFirstAndSecondRead;
}
/**
- * Returns the read footer metrics for all file types.
+ * Returns the file type.
*
- * @return the read footer metrics as a string
+ * @return the file type
*/
- @Override
- public String toString() {
- StringBuilder readFooterMetric = new StringBuilder();
- appendMetrics(readFooterMetric, PARQUET);
- appendMetrics(readFooterMetric, NON_PARQUET);
- return readFooterMetric.toString();
+ private FileType getFileType() {
+ return fileType;
+ }
+ }
+
+ /**
+ * Constructor to initialize the IOStatisticsStore with counters and mean statistics.
+ */
+ public AbfsReadFooterMetrics() {
+ IOStatisticsStore ioStatisticsStore = iostatisticsStore()
+ .withCounters(getMetricNames(TYPE_COUNTER))
+ .withMeanStatistics(getMetricNames(TYPE_MEAN))
+ .build();
+ setIOStatistics(ioStatisticsStore);
+ }
+
+ /**
+ * Constructor to initialize the IOStatisticsStore with counters and mean statistics,
+ * and a map to track file type metrics.
+ *
+ * @param fileTypeMetricsMap the map to track file type metrics
+ */
+ public AbfsReadFooterMetrics(Map<String, FileTypeMetrics> fileTypeMetricsMap) {
Review Comment:
we can try combining the 2 constructors from the above comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647783790
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -372,19 +436,26 @@ public AbfsClient(final URL baseUrl,
final SASTokenProvider sasTokenProvider,
final EncryptionContextProvider encryptionContextProvider,
final AbfsClientContext abfsClientContext,
- final AbfsServiceType abfsServiceType)
+ final String fileSystemId, final AbfsServiceType abfsServiceType)
throws IOException {
this(baseUrl, sharedKeyCredentials, abfsConfiguration,
- encryptionContextProvider, abfsClientContext, abfsServiceType);
+ encryptionContextProvider, abfsClientContext, fileSystemId, abfsServiceType);
this.sasTokenProvider = sasTokenProvider;
this.tokenProvider = tokenProvider;
}
@Override
public void close() throws IOException {
- if (isMetricCollectionEnabled && runningTimerTask != null) {
- runningTimerTask.cancel();
- timer.cancel();
+ if (isMetricCollectionEnabled()) {
Review Comment:
There are few tests which already cover this. Will check if more test cases are needed to cover more scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647752425
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java:
##########
@@ -217,11 +233,13 @@ private AbfsClient(final URL baseUrl,
final AbfsConfiguration abfsConfiguration,
final EncryptionContextProvider encryptionContextProvider,
final AbfsClientContext abfsClientContext,
+ final String fileSystemId,
Review Comment:
Yes, make sense. Will do this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anujmodi2021 (via GitHub)" <gi...@apache.org>.
anujmodi2021 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647989027
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
+ public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
Review Comment:
Yes. I think we should have common prefix for all the metric related configs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "anmolanmol1234 (via GitHub)" <gi...@apache.org>.
anmolanmol1234 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2647838292
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBackoffMetrics.java:
##########
@@ -305,7 +328,9 @@ public String toString() {
return EMPTY_STRING;
}
StringBuilder metricBuilder = new StringBuilder();
- getRetryMetrics(metricBuilder);
+ if (isRetryMetricEnabled) {
Review Comment:
I think we should always have retry metrics added as a part of these aggregate metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] Hadoop-19676: ABFS: Aggregated Metrics Manager with Multiple Emission Criteria [hadoop]
Posted by "bhattmanish98 (via GitHub)" <gi...@apache.org>.
bhattmanish98 commented on code in PR #8137:
URL: https://github.com/apache/hadoop/pull/8137#discussion_r2660838193
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java:
##########
@@ -136,6 +136,13 @@ public final class FileSystemConfigurations {
public static final boolean DEFAULT_ENABLE_AUTOTHROTTLING = false;
public static final int DEFAULT_METRIC_IDLE_TIMEOUT_MS = 60_000;
public static final int DEFAULT_METRIC_ANALYSIS_TIMEOUT_MS = 60_000;
+ public static final boolean DEFAULT_METRICS_COLLECTION_ENABLED = true;
+ public static final boolean DEFAULT_SHOULD_EMIT_METRICS_ON_IDLE_TIME = false;
+ public static final long DEFAULT_METRICS_EMIT_THRESHOLD = 100_000L;
+ public static final long DEFAULT_METRICS_EMIT_THRESHOLD_INTERVAL_SECS = 60;
Review Comment:
As discussed, no changes required here.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsReadFooterMetrics.java:
##########
@@ -62,476 +63,536 @@
* This class is responsible for tracking and updating metrics related to reading footers in files.
*/
public class AbfsReadFooterMetrics extends AbstractAbfsStatisticsSource {
- private static final Logger LOG = LoggerFactory.getLogger(AbfsReadFooterMetrics.class);
- private static final String FOOTER_LENGTH = "20";
- private static final List<FileType> FILE_TYPE_LIST =
- Arrays.asList(FileType.values());
- private final Map<String, FileTypeMetrics> fileTypeMetricsMap =
- new ConcurrentHashMap<>();
+
+ private static final Logger LOG = LoggerFactory.getLogger(
+ AbfsReadFooterMetrics.class);
+
Review Comment:
This line was added by the formatter we follow in OSS repo.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ this.rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleAtFixedRate(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
+ }));
+ }
+
+ /**
+ * Get the singleton instance of AggregateMetricsManager.
+ *
+ * @param dispatchIntervalInMins Interval in minutes for dispatching metrics.
+ * @param permitsPerSecond Rate limit for dispatching metrics.
+ * @return Singleton instance of AggregateMetricsManager.
+ */
+ public static AggregateMetricsManager get(final long dispatchIntervalInMins,
Review Comment:
Renamed it to getInstance
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java:
##########
@@ -876,6 +894,308 @@ public void testIsNonEmptyDirectory() throws IOException {
false, 1, true);
}
+ /**
+ * Test to verify that in case metric account is not set,
+ * metric collection is enabled with default metric format
+ * and account url.
+ *
+ * @throws Exception in case of any failure
+ */
+ @Test
+ public void testMetricAccountFallback() throws Exception {
+ Configuration configuration = getRawConfiguration();
+ configuration.setBoolean(
+ AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION, false);
+ configuration.setBoolean(FS_AZURE_ACCOUNT_IS_HNS_ENABLED, false);
+ configuration.unset(FS_AZURE_METRIC_ACCOUNT_NAME);
+ configuration.unset(FS_AZURE_METRIC_ACCOUNT_KEY);
+ configuration.unset(FS_AZURE_METRIC_FORMAT);
+ configuration.setBoolean(FS_AZURE_ALWAYS_USE_HTTPS, true);
+ final AzureBlobFileSystem fs = getFileSystem(configuration);
+ Assertions.assertThat(
+ fs.getAbfsStore().getAbfsConfiguration().getMetricFormat())
+ .describedAs(
+ "In case metric format is not set, metric format should "
+ + "be defaulted to internal metric format")
+ .isEqualTo(MetricFormat.INTERNAL_METRIC_FORMAT);
+
+ Assertions.assertThat(
+ fs.getAbfsStore().getClient().isMetricCollectionEnabled())
+ .describedAs(
+ "Metric collection should be enabled even if metric account is not set")
+ .isTrue();
+
+ Assertions.assertThat(
Review Comment:
Yes, but there is no backend call it has done yet to make backoff counter or read footer counter nonzero, so no metrics will be emitted.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java:
##########
@@ -71,7 +71,14 @@ public final class ConfigurationKeys {
public static final String FS_AZURE_ACCOUNT_KEY_PROPERTY_NAME = "fs.azure.account.key";
public static final String FS_AZURE_METRIC_ACCOUNT_NAME = "fs.azure.metric.account.name";
public static final String FS_AZURE_METRIC_ACCOUNT_KEY = "fs.azure.metric.account.key";
- public static final String FS_AZURE_METRIC_URI = "fs.azure.metric.uri";
+ public static final String FS_AZURE_METRIC_FORMAT = "fs.azure.metric.format";
Review Comment:
Fixed
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/MetricsBucket.java:
##########
@@ -0,0 +1,196 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * MetricsBucket holds metrics for multiple AbfsClients and
+ * dispatches them in batches, respecting rate limits.
+ */
+final class MetricsBucket {
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private final SimpleRateLimiter rateLimiter;
+
+ // Buffer to hold metrics before sending.
+ private final AtomicReference<ConcurrentLinkedQueue<String>> metricsBuffer =
+ new AtomicReference<>(new ConcurrentLinkedQueue<>());
+
+ // Set of registered AbfsClients.
+ private final Set<AbfsClient> clients =
+ ConcurrentHashMap.newKeySet();
+
+ // Maximum size of metrics header in characters.
+ private static final long MAX_HEADER_SIZE = 1024;
+
+ // Constructor
+ MetricsBucket(SimpleRateLimiter rateLimiter) {
+ this.rateLimiter = rateLimiter;
+ }
+
+ /**
+ * Register a new AbfsClient.
+ * @param client the AbfsClient to register
+ */
+ public void registerClient(AbfsClient client) {
+ if (client != null) {
+ clients.add(client);
+ }
+ }
+
+ /**
+ * Deregister an AbfsClient. If this is the last client, drain and send
+ * any remaining metrics.
+ * @param client the AbfsClient to deregister
+ * @return true if the client was deregistered, false otherwise
+ */
+ public boolean deregisterClient(AbfsClient client) {
+ if (client == null) {
+ return false;
+ }
+ ConcurrentLinkedQueue<String> batchToSend = null;
+ boolean isLastClient = false;
+
+ synchronized (this) {
+ if (!clients.contains(client)) {
+ return false;
+ }
+
+ if (clients.size() == 1) {
Review Comment:
As discussed, this change is not needed as the current code guarantees the metrics emission correctly.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
Review Comment:
The reason for sending the entire AbfsConfiguration was that we may need additional configuration values in the future; in that case, we can fetch them directly without having to change this again.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
+ }
+ }
+
+ /**
+ * Initialize the read footer metrics.
+ * In case the metrics are already initialized,
+ * create a new instance with the existing map.
Review Comment:
After the collected metrics are pushed to the backend, the counters are refreshed by creating new instances of the backoff and read-footer metrics rather than clearing each counter individually. The old instances are automatically garbage-collected.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
+
+ // Map of account name to MetricsBucket.
+ private final ConcurrentHashMap<String, MetricsBucket> buckets =
+ new ConcurrentHashMap<>();
+
+ // Scheduler for periodic dispatching of metrics.
+ private final ScheduledExecutorService scheduler;
+
+ // Private constructor to enforce singleton pattern.
+ private AggregateMetricsManager(final long dispatchIntervalInMins,
+ final int permitsPerSecond) throws InvalidConfigurationValueException {
+
+ if (dispatchIntervalInMins <= 0) {
+ throw new InvalidConfigurationValueException(
+ "dispatchIntervalInMins must be > 0");
+ }
+
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+
+ this.rateLimiter = new SimpleRateLimiter(permitsPerSecond);
+
+ // Initialize scheduler for periodic dispatching of metrics.
+ this.scheduler = Executors.newSingleThreadScheduledExecutor(r -> {
+ Thread t = new Thread(r, "ABFS-Aggregated-Metrics-Dispatcher");
+ t.setDaemon(true);
+ return t;
+ });
+
+ // Schedule periodic dispatching of metrics.
+ this.scheduler.scheduleAtFixedRate(
+ this::dispatchMetrics,
+ dispatchIntervalInMins,
+ dispatchIntervalInMins,
+ TimeUnit.MINUTES);
+
+ // Add shutdown hook to dispatch remaining metrics on JVM shutdown.
+ Runtime.getRuntime().addShutdownHook(new Thread(() -> {
+ dispatchMetrics();
+ scheduler.shutdown();
Review Comment:
Since it is a sequential operation so shutdown will happen only after dispatchMetrics is completed.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsCountersImpl.java:
##########
@@ -201,20 +201,39 @@ public void initializeWriteResourceUtilizationMetrics() {
@Override
- public void initializeMetrics(MetricFormat metricFormat) {
+ public void initializeMetrics(final MetricFormat metricFormat,
+ final AbfsConfiguration abfsConfiguration) {
switch (metricFormat) {
- case INTERNAL_BACKOFF_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- break;
- case INTERNAL_FOOTER_METRIC_FORMAT:
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- case INTERNAL_METRIC_FORMAT:
- abfsBackoffMetrics = new AbfsBackoffMetrics();
- abfsReadFooterMetrics = new AbfsReadFooterMetrics();
- break;
- default:
- break;
+ case INTERNAL_BACKOFF_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ break;
+ case INTERNAL_FOOTER_METRIC_FORMAT:
+ initializeReadFooterMetrics();
+ case INTERNAL_METRIC_FORMAT:
+ abfsBackoffMetrics = new AbfsBackoffMetrics(
+ abfsConfiguration.isBackoffRetryMetricsEnabled());
+ initializeReadFooterMetrics();
+ break;
+ default:
+ break;
+ }
+ }
+
+ /**
+ * Initialize the read footer metrics.
+ * In case the metrics are already initialized,
+ * create a new instance with the existing map.
+ */
+ private void initializeReadFooterMetrics() {
+ if (abfsReadFooterMetrics == null) {
+ abfsReadFooterMetrics = new AbfsReadFooterMetrics();
+ } else {
+ //In case metrics is emitted based on total count, there could be a chance
+ // that file type for which we have calculated the type will be lost.
+ // To avoid that, creating a new instance with existing map.
+ abfsReadFooterMetrics = new AbfsReadFooterMetrics(
Review Comment:
Here we are creating new object of AbfsReadFooterMetrics. Thats why we are copying the file type information to new object.
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsReadFooterMetrics.java:
##########
@@ -57,32 +54,14 @@
import org.apache.hadoop.fs.azurebfs.utils.TracingHeaderValidator;
import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
-import static org.assertj.core.api.Assumptions.assumeThat;
-
public class ITestAbfsReadFooterMetrics extends AbstractAbfsScaleTest {
- public ITestAbfsReadFooterMetrics() throws Exception {
- checkPrerequisites();
- }
-
- private void checkPrerequisites(){
Review Comment:
Now, it is not required as irrespective of these configurations, we will the metrics in the main account.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/SimpleRateLimiter.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.util.concurrent.locks.LockSupport;
+
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+
+public final class SimpleRateLimiter {
+
+ // Interval between permits in nanoseconds.
+ private final long intervalNanos;
+
+ // Next allowed time to acquire a permit in nanoseconds.
+ private long nextAllowedTime;
+
+ /** Number of nanoseconds in one second. */
+ private static final long NANOS_PER_SECOND = 1_000_000_000L;
+
+ /**
+ * Constructs a SimpleRateLimiter that allows the specified number of
+ * permits per second.
+ *
+ * @param permitsPerSecond Number of permits allowed per second.
+ * @throws InvalidConfigurationValueException if permitsPerSecond is
+ * less than or equal to zero.
+ */
+ public SimpleRateLimiter(int permitsPerSecond)
+ throws InvalidConfigurationValueException {
+ if (permitsPerSecond <= 0) {
+ throw new InvalidConfigurationValueException(
+ "permitsPerSecond must be > 0");
+ }
+ this.intervalNanos = NANOS_PER_SECOND / permitsPerSecond;
+ this.nextAllowedTime = System.nanoTime();
+ }
+
+ /**
+ * Acquires a permit from the rate limiter, blocking until one is available.
+ */
+ public synchronized void acquire() {
Review Comment:
Made the changes to make it time bounded wait.
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AggregateMetricsManager.java:
##########
@@ -0,0 +1,175 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.services;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException;
+import org.apache.hadoop.fs.azurebfs.utils.SimpleRateLimiter;
+
+/**
+ * AggregateMetricsManager manages metrics collection and dispatching
+ * for multiple AbfsClients across different accounts.
+ */
+public final class AggregateMetricsManager {
+
+ // Singleton instance of AggregateMetricsManager.
+ private static volatile AggregateMetricsManager instance;
+
+ // Rate limiter to control the rate of dispatching metrics.
+ private static volatile SimpleRateLimiter rateLimiter;
Review Comment:
Yes, this is what we want as well - Ratelimit on the JVM level so that there should be less number of metrics calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org