You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "aparna0522 (via GitHub)" <gi...@apache.org> on 2025/10/27 18:19:10 UTC
[PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
aparna0522 opened a new pull request, #4445:
URL: https://github.com/apache/cassandra/pull/4445
We are adding new metrics to make the Mutation Tracking functionality easy to operate.
Metrics Added:
- Offsets applied via broadcast and write time
- Read summary sizes
- Number of unreconciled mutations
- Size of MutationJournal on disk
patch by Aparna Naik; reviewed by <Reviewers> for CASSANDRA-20986
Thanks for sending a pull request! Here are some tips if you're new here:
* Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
* Be sure to keep the PR description updated to reflect all changes.
* Write your PR title to summarize what this PR proposes.
* If possible, provide a concise example to reproduce the issue for a faster review.
* Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
* If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
Commit messages should follow the following format:
```
<One sentence description, usually Jira title or CHANGES.txt summary>
<Optional lengthier description (context on patch)>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "aratno (via GitHub)" <gi...@apache.org>.
aratno closed pull request #4445: CASSANDRA-20986: Add metrics for Mutation Tracking
URL: https://github.com/apache/cassandra/pull/4445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "bdeggleston (via GitHub)" <gi...@apache.org>.
bdeggleston commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2496437278
##########
src/java/org/apache/cassandra/replication/CoordinatorLog.java:
##########
@@ -220,6 +227,9 @@ private void updateWitnessedReplicatedOffsets(Offsets offsets, int onNodeId)
}
}
});
+
+ // Record metric for newly witnessed offsets only
+ MutationTrackingMetrics.instance.broadcastOffsetsDiscovered.inc(newlyWitnessedCount[0]);
Review Comment:
the other consideration is allocations. In this case though, this doesn't happen on a read or write path so it's not going to be happening thousands of times per second so it's fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "aparna0522 (via GitHub)" <gi...@apache.org>.
aparna0522 commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2492137615
##########
src/java/org/apache/cassandra/replication/CoordinatorLog.java:
##########
@@ -220,6 +227,9 @@ private void updateWitnessedReplicatedOffsets(Offsets offsets, int onNodeId)
}
}
});
+
+ // Record metric for newly witnessed offsets only
+ MutationTrackingMetrics.instance.broadcastOffsetsDiscovered.inc(newlyWitnessedCount[0]);
Review Comment:
I think the bottom line is one atomic operation vs multiple. If addAll() typically invokes the consumer just a few times, then, the difference is negligible, in which case I can see the argument for cleaner code without the array.
Additionally, I tried to check how other metrics are being collected and I see an accumulate-then-increment pattern, which is similar to the current implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "aparna0522 (via GitHub)" <gi...@apache.org>.
aparna0522 commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2491897495
##########
src/java/org/apache/cassandra/journal/Journal.java:
##########
@@ -832,6 +832,32 @@ private void shutDown() throws InterruptedException
}
}
+ public long getDiskSpaceUsed()
+ {
+ long totalSize = 0;
+
+ try (ReferencedSegments<K, V> refs = selectAndReference(s -> true))
+ {
+ for (Segment<K, V> segment : refs.all())
+ {
+ File dataFile = segment.descriptor.fileFor(Component.DATA);
+ if (dataFile.exists())
+ totalSize += dataFile.length();
+ }
+ }
+
+ // Add current active segment
+ ActiveSegment<K, V> active = currentSegment;
Review Comment:
I believe you are correct... I traced through the code and I see that `selectAndReference(s -> true)` already includes the active segment. When a new active segment is created in `advanceSegment()`, it calls `addNewActiveSegment(currentSegment = availableSegment)` which is adding the active segment to the segments collection from `Segments.withNewActiveSegment()` method.
Also, since I am using the predicate `s -> true`, it ensures that activeSegment is a part of it.
Thanks for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "aparna0522 (via GitHub)" <gi...@apache.org>.
aparna0522 commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2491897495
##########
src/java/org/apache/cassandra/journal/Journal.java:
##########
@@ -832,6 +832,32 @@ private void shutDown() throws InterruptedException
}
}
+ public long getDiskSpaceUsed()
+ {
+ long totalSize = 0;
+
+ try (ReferencedSegments<K, V> refs = selectAndReference(s -> true))
+ {
+ for (Segment<K, V> segment : refs.all())
+ {
+ File dataFile = segment.descriptor.fileFor(Component.DATA);
+ if (dataFile.exists())
+ totalSize += dataFile.length();
+ }
+ }
+
+ // Add current active segment
+ ActiveSegment<K, V> active = currentSegment;
Review Comment:
I believe you are correct... I traced through the code and I see that `selectAndReference(s -> true)` already includes the active segment. When a new active segment is created, it calls `addNewActiveSegment(currentSegment = availableSegment)` which is adding the active segment to the segments collection, Also, since I am using the predicate `s -> true` does ensure that activeSegment is a part of it.
Thanks for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "bdeggleston (via GitHub)" <gi...@apache.org>.
bdeggleston commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2488152350
##########
src/java/org/apache/cassandra/replication/CoordinatorLog.java:
##########
@@ -220,6 +227,9 @@ private void updateWitnessedReplicatedOffsets(Offsets offsets, int onNodeId)
}
}
});
+
+ // Record metric for newly witnessed offsets only
+ MutationTrackingMetrics.instance.broadcastOffsetsDiscovered.inc(newlyWitnessedCount[0]);
Review Comment:
I'm on the fence about whether it's better to allocate the array and call this once, or just call increment for each range in the range consumer. WDYT?
##########
src/java/org/apache/cassandra/journal/Journal.java:
##########
@@ -832,6 +832,32 @@ private void shutDown() throws InterruptedException
}
}
+ public long getDiskSpaceUsed()
+ {
+ long totalSize = 0;
+
+ try (ReferencedSegments<K, V> refs = selectAndReference(s -> true))
+ {
+ for (Segment<K, V> segment : refs.all())
+ {
+ File dataFile = segment.descriptor.fileFor(Component.DATA);
+ if (dataFile.exists())
+ totalSize += dataFile.length();
+ }
+ }
+
+ // Add current active segment
+ ActiveSegment<K, V> active = currentSegment;
Review Comment:
I _think_ that segments will reference all segments, including the active segment, so we'd be double counting here. I'm only like 95% sure that's the case though, so definitely dig into this and make sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "aparna0522 (via GitHub)" <gi...@apache.org>.
aparna0522 commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2492137615
##########
src/java/org/apache/cassandra/replication/CoordinatorLog.java:
##########
@@ -220,6 +227,9 @@ private void updateWitnessedReplicatedOffsets(Offsets offsets, int onNodeId)
}
}
});
+
+ // Record metric for newly witnessed offsets only
+ MutationTrackingMetrics.instance.broadcastOffsetsDiscovered.inc(newlyWitnessedCount[0]);
Review Comment:
I think the bottom line is one atomic operation vs multiple. If addAll() typically invokes the consumer just a few times, then, the difference is negligible, in which case I can see the argument for cleaner code without the array.
However, I tried to check how other metrics are being collected and I see an accumulate-then-increment pattern, which is similar to the current implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-20986: Add metrics for Mutation Tracking [cassandra]
Posted by "aratno (via GitHub)" <gi...@apache.org>.
aratno commented on code in PR #4445:
URL: https://github.com/apache/cassandra/pull/4445#discussion_r2585866514
##########
src/java/org/apache/cassandra/metrics/MutationTrackingMetrics.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.metrics;
+
+import com.codahale.metrics.Counter;
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Histogram;
+import org.apache.cassandra.replication.MutationJournal;
+import org.apache.cassandra.replication.MutationTrackingService;
+
+import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
+
+public class MutationTrackingMetrics
+{
+ public static final String TYPE_NAME = "MutationTracking";
+ private static final MetricNameFactory factory = new DefaultNameFactory(TYPE_NAME);
+
+ public static final MutationTrackingMetrics instance = new MutationTrackingMetrics();
+
+ public final Counter broadcastOffsetsDiscovered; // Newly-witnessed offsets discovered via broadcast
+ public final Counter writeTimeOffsetsDiscovered; // Newly-witnessed offsets discovered at write time
+ public final Histogram readSummarySize; // Read summary sizes
+ public final Gauge<Long> unreconciledMutationCount; // Number of unreconciled mutations
+ public final Gauge<Long> journalDiskSpaceUsed; // Size of MutationJournal on disk
+
+ @SuppressWarnings("Convert2MethodRef")
+ private MutationTrackingMetrics()
+ {
+ broadcastOffsetsDiscovered = Metrics.counter(factory.createMetricName("broadcastOffsetsDiscovered"));
+ writeTimeOffsetsDiscovered = Metrics.counter(factory.createMetricName("WriteTimeOffsetsDiscovered"));
+ readSummarySize = Metrics.histogram(factory.createMetricName("ReadSummarySize"), false);
Review Comment:
Might be clearer to have considerZeroes true, to differentiate between empty and non-empty summaries
##########
src/java/org/apache/cassandra/metrics/MutationTrackingMetrics.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.metrics;
+
+import com.codahale.metrics.Counter;
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.Histogram;
+import org.apache.cassandra.replication.MutationJournal;
+import org.apache.cassandra.replication.MutationTrackingService;
+
+import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
+
+public class MutationTrackingMetrics
+{
+ public static final String TYPE_NAME = "MutationTracking";
+ private static final MetricNameFactory factory = new DefaultNameFactory(TYPE_NAME);
+
+ public static final MutationTrackingMetrics instance = new MutationTrackingMetrics();
+
+ public final Counter broadcastOffsetsDiscovered; // Newly-witnessed offsets discovered via broadcast
+ public final Counter writeTimeOffsetsDiscovered; // Newly-witnessed offsets discovered at write time
+ public final Histogram readSummarySize; // Read summary sizes
+ public final Gauge<Long> unreconciledMutationCount; // Number of unreconciled mutations
+ public final Gauge<Long> journalDiskSpaceUsed; // Size of MutationJournal on disk
+
+ @SuppressWarnings("Convert2MethodRef")
+ private MutationTrackingMetrics()
+ {
+ broadcastOffsetsDiscovered = Metrics.counter(factory.createMetricName("broadcastOffsetsDiscovered"));
Review Comment:
Nit
```suggestion
broadcastOffsetsDiscovered = Metrics.counter(factory.createMetricName("BroadcastOffsetsDiscovered"));
```
##########
src/java/org/apache/cassandra/replication/MutationTrackingService.java:
##########
@@ -330,7 +331,9 @@ public MutationSummary createSummaryForKey(DecoratedKey key, TableId tableId, bo
shardLock.readLock().lock();
try
{
- return getOrCreateShards(tableId).createSummaryForKey(key, tableId, includePending);
+ MutationSummary summary = getOrCreateShards(tableId).createSummaryForKey(key, tableId, includePending);
+ MutationTrackingMetrics.instance.readSummarySize.update(summary.size());
Review Comment:
Should also apply to createSummaryForRange, so we measure summary sizes for range reads as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org