You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2022/07/28 00:34:01 UTC
[GitHub] [cassandra] frankgh commented on a diff in pull request #1754: 17725 4.1
frankgh commented on code in PR #1754:
URL: https://github.com/apache/cassandra/pull/1754#discussion_r931664189
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -252,6 +253,21 @@ public LongBytesPerSecondBound(long bytesPerSecond)
{
this(bytesPerSecond, BYTES_PER_SECOND);
}
+
+ // this one should be used only for backward compatibility for stream_throughput_outbound and inter_dc_stream_throughput_outbound
+ // which were in megabits per second in 4.0. Do not start using it for any new properties
+ public static LongBytesPerSecondBound megabitsPerSecondInBytesPerSecond(long megabitsPerSecond)
+ {
+ final double BYTES_PER_MEGABIT = 125_000;
+ double bytesPerSecond = (double) megabitsPerSecond * BYTES_PER_MEGABIT;
Review Comment:
Why use double here? long should be fine
```suggestion
final long BYTES_PER_MEGABIT = 125_000L;
long bytesPerSecond = megabitsPerSecond * BYTES_PER_MEGABIT;
```
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -252,6 +253,21 @@ public LongBytesPerSecondBound(long bytesPerSecond)
{
this(bytesPerSecond, BYTES_PER_SECOND);
}
+
+ // this one should be used only for backward compatibility for stream_throughput_outbound and inter_dc_stream_throughput_outbound
+ // which were in megabits per second in 4.0. Do not start using it for any new properties
+ public static LongBytesPerSecondBound megabitsPerSecondInBytesPerSecond(long megabitsPerSecond)
Review Comment:
should we annotate with `@Deprecated` if this is not intended to be used
##########
src/java/org/apache/cassandra/service/StorageServiceMBean.java:
##########
@@ -625,26 +625,71 @@ default int upgradeSSTables(String keyspaceName, boolean excludeCurrentVersion,
public long getTruncateRpcTimeout();
public void setStreamThroughputMbitPerSec(int value);
+ /**
+ * Please use the new getStreamThroughputMbitPerSecAsDouble method as this one will provide a rounded value
+ * @return stream_throughput_outbound in megabits
Review Comment:
maybe used the deprecated annotation for javadocs here?
```suggestion
* @return stream_throughput_outbound in megabits
* @deprecated Use getStreamThroughputMbitPerSecAsDouble instead
```
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -212,7 +213,7 @@ public boolean equals(Object obj)
@Override
public String toString()
{
- return Math.round(quantity) + unit.symbol;
+ return DoubleMath.isMathematicalInteger(quantity) ? ((long)quantity + unit.symbol) : (quantity + unit.symbol);
Review Comment:
this changes behavior. `Math.round` of `999.999` will return -> `1000`. Whereas, `(long)999.999` returns `999`. Is this intended?
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -252,6 +253,21 @@ public LongBytesPerSecondBound(long bytesPerSecond)
{
this(bytesPerSecond, BYTES_PER_SECOND);
}
+
+ // this one should be used only for backward compatibility for stream_throughput_outbound and inter_dc_stream_throughput_outbound
+ // which were in megabits per second in 4.0. Do not start using it for any new properties
+ public static LongBytesPerSecondBound megabitsPerSecondInBytesPerSecond(long megabitsPerSecond)
+ {
+ final double BYTES_PER_MEGABIT = 125_000;
+ double bytesPerSecond = (double) megabitsPerSecond * BYTES_PER_MEGABIT;
+
+ if (megabitsPerSecond >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid data rate: " + megabitsPerSecond + " megabits per second; " +
+ "stream_throughput_outbound and inter_dc_stream_throughput_outbound" +
+ " should be between 0 and " + Integer.MAX_VALUE + " in megabits per second");
+
+ return new LongBytesPerSecondBound(bytesPerSecond, BYTES_PER_SECOND);
Review Comment:
`LongBytesPerSecondBound`'s constructor takes a double, but it should really take a long. We cast from double to long internally. I tried making `DataRateSpec`'s `quantity` a `long`, and tests are passing.
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -252,6 +253,21 @@ public LongBytesPerSecondBound(long bytesPerSecond)
{
this(bytesPerSecond, BYTES_PER_SECOND);
}
+
+ // this one should be used only for backward compatibility for stream_throughput_outbound and inter_dc_stream_throughput_outbound
+ // which were in megabits per second in 4.0. Do not start using it for any new properties
+ public static LongBytesPerSecondBound megabitsPerSecondInBytesPerSecond(long megabitsPerSecond)
+ {
+ final double BYTES_PER_MEGABIT = 125_000;
Review Comment:
should we declare it as a field constant instead?
##########
src/java/org/apache/cassandra/config/Config.java:
##########
@@ -315,7 +315,7 @@ public MemtableOptions()
public Integer unlogged_batch_across_partitions_warn_threshold = 10;
public volatile Integer concurrent_compactors;
@Replaces(oldName = "compaction_throughput_mb_per_sec", converter = Converters.MEBIBYTES_PER_SECOND_DATA_RATE, deprecated = true)
- public volatile DataRateSpec.IntMebibytesPerSecondBound compaction_throughput = new DataRateSpec.IntMebibytesPerSecondBound("16MiB/s");
Review Comment:
do we want to keep the `DataRateSpec.IntMebibytesPerSecondBound` class? I only see it being used in tests. Maybe remove early before someone starts relying on it.
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -252,6 +253,21 @@ public LongBytesPerSecondBound(long bytesPerSecond)
{
this(bytesPerSecond, BYTES_PER_SECOND);
}
+
+ // this one should be used only for backward compatibility for stream_throughput_outbound and inter_dc_stream_throughput_outbound
+ // which were in megabits per second in 4.0. Do not start using it for any new properties
+ public static LongBytesPerSecondBound megabitsPerSecondInBytesPerSecond(long megabitsPerSecond)
+ {
+ final double BYTES_PER_MEGABIT = 125_000;
+ double bytesPerSecond = (double) megabitsPerSecond * BYTES_PER_MEGABIT;
+
+ if (megabitsPerSecond >= Integer.MAX_VALUE)
Review Comment:
Did you mean `>` here?
```suggestion
if (megabitsPerSecond > Integer.MAX_VALUE)
```
##########
src/java/org/apache/cassandra/config/DataRateSpec.java:
##########
@@ -385,7 +386,7 @@ public double toMegabitsPerSecond(double d)
{
if (d > MAX / (MEGABITS_PER_MEBIBYTE))
return MAX;
- return Math.round(d * MEGABITS_PER_MEBIBYTE);
+ return d * MEGABITS_PER_MEBIBYTE;
Review Comment:
this also changes behavior but it's probably desired
##########
src/java/org/apache/cassandra/config/DatabaseDescriptor.java:
##########
@@ -2016,59 +2055,115 @@ public static int getStreamThroughputOutboundMegabitsPerSec()
return conf.stream_throughput_outbound.toMegabitsPerSecondAsInt();
}
+ public static double getStreamThroughputOutboundMegabitsPerSecAsDouble()
+ {
+ return conf.stream_throughput_outbound.toMegabitsPerSecond();
+ }
+
public static double getStreamThroughputOutboundMebibytesPerSec()
{
return conf.stream_throughput_outbound.toMebibytesPerSecond();
}
- public static void setStreamThroughputOutboundMegabitsPerSec(int value)
+ public static double getStreamThroughputOutboundBytesPerSec()
{
- conf.stream_throughput_outbound = DataRateSpec.IntMebibytesPerSecondBound.megabitsPerSecondInMebibytesPerSecond(value);
+ return conf.stream_throughput_outbound.toBytesPerSecond();
}
- public static int getEntireSSTableStreamThroughputOutboundMebibytesPerSecAsInt()
+ public static int getStreamThroughputOutboundMebibytesPerSecAsInt()
{
- return conf.entire_sstable_stream_throughput_outbound.toMebibytesPerSecondAsInt();
+ return conf.stream_throughput_outbound.toMebibytesPerSecondAsInt();
+ }
+
+ public static void setStreamThroughputOutboundMebibytesPerSecAsInt(int value)
+ {
+ if (MEBIBYTES_PER_SECOND.toMegabitsPerSecond(value) >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid value of stream_throughput_outbound: " + value);
+
+ long valueInBytes = value * 1024L * 1024L;
+ conf.stream_throughput_outbound = new DataRateSpec.LongBytesPerSecondBound(valueInBytes);
+ }
+
+ public static void setStreamThroughputOutboundMegabitsPerSec(int value)
+ {
+ conf.stream_throughput_outbound = DataRateSpec.LongBytesPerSecondBound.megabitsPerSecondInBytesPerSecond(value);
}
public static double getEntireSSTableStreamThroughputOutboundMebibytesPerSec()
{
return conf.entire_sstable_stream_throughput_outbound.toMebibytesPerSecond();
}
+ public static double getEntireSSTableStreamThroughputOutboundBytesPerSec()
+ {
+ return conf.entire_sstable_stream_throughput_outbound.toBytesPerSecond();
+ }
+
public static void setEntireSSTableStreamThroughputOutboundMebibytesPerSec(int value)
{
- conf.entire_sstable_stream_throughput_outbound = new DataRateSpec.IntMebibytesPerSecondBound(value);
+ if (MEBIBYTES_PER_SECOND.toMebibytesPerSecond(value) >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid value of entire_sstable_stream_throughput_outbound: " + value);
+
+ long valueInBytes = value * 1024L * 1024L;
+ conf.entire_sstable_stream_throughput_outbound = new DataRateSpec.LongBytesPerSecondBound(valueInBytes);
}
public static int getInterDCStreamThroughputOutboundMegabitsPerSec()
{
return conf.inter_dc_stream_throughput_outbound.toMegabitsPerSecondAsInt();
}
+ public static double getInterDCStreamThroughputOutboundMegabitsPerSecAsDouble()
+ {
+ return conf.inter_dc_stream_throughput_outbound.toMegabitsPerSecond();
+ }
+
public static double getInterDCStreamThroughputOutboundMebibytesPerSec()
{
return conf.inter_dc_stream_throughput_outbound.toMebibytesPerSecond();
}
+ public static double getInterDCStreamThroughputOutboundBytesPerSec()
+ {
+ return conf.inter_dc_stream_throughput_outbound.toBytesPerSecond();
+ }
+
+ public static int getInterDCStreamThroughputOutboundMebibytesPerSecAsInt()
+ {
+ return conf.inter_dc_stream_throughput_outbound.toMebibytesPerSecondAsInt();
+ }
+
+ public static void setInterDCStreamThroughputOutboundMebibytesPerSecAsInt(int value)
+ {
+ if (MEBIBYTES_PER_SECOND.toMegabitsPerSecond(value) >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid value of inter_dc_stream_throughput_outbound: " + value);
+
+ long valueInBytes = value * 1024L * 1024L;
+ conf.inter_dc_stream_throughput_outbound = new DataRateSpec.LongBytesPerSecondBound(valueInBytes);
+ }
+
public static void setInterDCStreamThroughputOutboundMegabitsPerSec(int value)
{
- conf.inter_dc_stream_throughput_outbound = DataRateSpec.IntMebibytesPerSecondBound.megabitsPerSecondInMebibytesPerSecond(value);
+ conf.inter_dc_stream_throughput_outbound = DataRateSpec.LongBytesPerSecondBound.megabitsPerSecondInBytesPerSecond(value);
}
- public static double getEntireSSTableInterDCStreamThroughputOutboundMebibytesPerSec()
+ public static double getEntireSSTableInterDCStreamThroughputOutboundBytesPerSec()
{
- return conf.entire_sstable_inter_dc_stream_throughput_outbound.toMebibytesPerSecond();
+ return conf.entire_sstable_inter_dc_stream_throughput_outbound.toBytesPerSecond();
}
- public static int getEntireSSTableInterDCStreamThroughputOutboundMebibytesPerSecAsInt()
+ public static double getEntireSSTableInterDCStreamThroughputOutboundMebibytesPerSec()
{
- return conf.entire_sstable_inter_dc_stream_throughput_outbound.toMebibytesPerSecondAsInt();
+ return conf.entire_sstable_inter_dc_stream_throughput_outbound.toMebibytesPerSecond();
}
public static void setEntireSSTableInterDCStreamThroughputOutboundMebibytesPerSec(int value)
{
- conf.entire_sstable_inter_dc_stream_throughput_outbound = new DataRateSpec.IntMebibytesPerSecondBound(value);
+ if (MEBIBYTES_PER_SECOND.toMebibytesPerSecond(value) >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid value of entire_sstable_inter_dc_stream_throughput_outbound: " + value);
+
+ long valueInBytes = value * 1024L * 1024L;
Review Comment:
It feels like we do this a sufficient number of times that it might be worth defining a constant
` static final int ONE_MEBIBTE_IN_BYTES = 1024L * 1024L;`
##########
src/java/org/apache/cassandra/config/DatabaseDescriptor.java:
##########
@@ -906,6 +898,36 @@ else if (conf.max_value_size.toMebibytes() >= 2048)
logInitializationOutcome(logger);
}
+ @VisibleForTesting
+ static void validateUpperBoundStreamingConfig() throws ConfigurationException
+ {
+ // below 2 checks are needed in order to match the pre-CASSANDRA-15234 upper bound for those parameters which were still in megabits per second
+ if (conf.stream_throughput_outbound.toMegabitsPerSecond() >= Integer.MAX_VALUE)
Review Comment:
I'm confused why we exclude `Integer.MAX_VALUE`. Should this be `(conf.stream_throughput_outbound.toMegabitsPerSecond() > Integer.MAX_VALUE)` instead?
##########
src/java/org/apache/cassandra/config/DatabaseDescriptor.java:
##########
@@ -3186,7 +3281,10 @@ public static int getCacheLoadTimeout()
@VisibleForTesting
public static void setCacheLoadTimeout(int seconds)
{
- conf.cache_load_timeout = new DurationSpec.IntSecondsBound(seconds);
+ if (seconds < 0)
Review Comment:
minor nit:
```suggestion
if (seconds <= 0)
```
##########
src/java/org/apache/cassandra/config/DatabaseDescriptor.java:
##########
@@ -2016,59 +2055,115 @@ public static int getStreamThroughputOutboundMegabitsPerSec()
return conf.stream_throughput_outbound.toMegabitsPerSecondAsInt();
}
+ public static double getStreamThroughputOutboundMegabitsPerSecAsDouble()
+ {
+ return conf.stream_throughput_outbound.toMegabitsPerSecond();
+ }
+
public static double getStreamThroughputOutboundMebibytesPerSec()
{
return conf.stream_throughput_outbound.toMebibytesPerSecond();
}
- public static void setStreamThroughputOutboundMegabitsPerSec(int value)
+ public static double getStreamThroughputOutboundBytesPerSec()
{
- conf.stream_throughput_outbound = DataRateSpec.IntMebibytesPerSecondBound.megabitsPerSecondInMebibytesPerSecond(value);
+ return conf.stream_throughput_outbound.toBytesPerSecond();
}
- public static int getEntireSSTableStreamThroughputOutboundMebibytesPerSecAsInt()
+ public static int getStreamThroughputOutboundMebibytesPerSecAsInt()
{
- return conf.entire_sstable_stream_throughput_outbound.toMebibytesPerSecondAsInt();
+ return conf.stream_throughput_outbound.toMebibytesPerSecondAsInt();
+ }
+
+ public static void setStreamThroughputOutboundMebibytesPerSecAsInt(int value)
+ {
+ if (MEBIBYTES_PER_SECOND.toMegabitsPerSecond(value) >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid value of stream_throughput_outbound: " + value);
+
+ long valueInBytes = value * 1024L * 1024L;
+ conf.stream_throughput_outbound = new DataRateSpec.LongBytesPerSecondBound(valueInBytes);
+ }
+
+ public static void setStreamThroughputOutboundMegabitsPerSec(int value)
+ {
+ conf.stream_throughput_outbound = DataRateSpec.LongBytesPerSecondBound.megabitsPerSecondInBytesPerSecond(value);
}
public static double getEntireSSTableStreamThroughputOutboundMebibytesPerSec()
{
return conf.entire_sstable_stream_throughput_outbound.toMebibytesPerSecond();
}
+ public static double getEntireSSTableStreamThroughputOutboundBytesPerSec()
+ {
+ return conf.entire_sstable_stream_throughput_outbound.toBytesPerSecond();
+ }
+
public static void setEntireSSTableStreamThroughputOutboundMebibytesPerSec(int value)
{
- conf.entire_sstable_stream_throughput_outbound = new DataRateSpec.IntMebibytesPerSecondBound(value);
+ if (MEBIBYTES_PER_SECOND.toMebibytesPerSecond(value) >= Integer.MAX_VALUE)
+ throw new IllegalArgumentException("Invalid value of entire_sstable_stream_throughput_outbound: " + value);
+
+ long valueInBytes = value * 1024L * 1024L;
+ conf.entire_sstable_stream_throughput_outbound = new DataRateSpec.LongBytesPerSecondBound(valueInBytes);
}
public static int getInterDCStreamThroughputOutboundMegabitsPerSec()
{
return conf.inter_dc_stream_throughput_outbound.toMegabitsPerSecondAsInt();
}
+ public static double getInterDCStreamThroughputOutboundMegabitsPerSecAsDouble()
+ {
+ return conf.inter_dc_stream_throughput_outbound.toMegabitsPerSecond();
+ }
+
public static double getInterDCStreamThroughputOutboundMebibytesPerSec()
{
return conf.inter_dc_stream_throughput_outbound.toMebibytesPerSecond();
}
+ public static double getInterDCStreamThroughputOutboundBytesPerSec()
+ {
+ return conf.inter_dc_stream_throughput_outbound.toBytesPerSecond();
+ }
+
+ public static int getInterDCStreamThroughputOutboundMebibytesPerSecAsInt()
Review Comment:
is this only used for JMX calls?
##########
src/java/org/apache/cassandra/tools/nodetool/GetCompactionThroughput.java:
##########
@@ -17,17 +17,34 @@
*/
package org.apache.cassandra.tools.nodetool;
+import com.google.common.math.DoubleMath;
+
import io.airlift.airline.Command;
+import io.airlift.airline.Option;
import org.apache.cassandra.tools.NodeProbe;
import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
-@Command(name = "getcompactionthroughput", description = "Print the MiB/s throughput cap for compaction in the system")
+@Command(name = "getcompactionthroughput", description = "Print the MiB/s throughput cap for compaction in the system as a rounded number")
public class GetCompactionThroughput extends NodeToolCmd
{
+ @SuppressWarnings("UnusedDeclaration")
+ @Option(name = { "-d", "--precise-mib" }, description = "Print the MiB/s throughput cap for compaction in the system as a precise number (double)")
+ private boolean compactionThroughputAsDouble;
+
@Override
public void execute(NodeProbe probe)
{
- probe.output().out.println("Current compaction throughput: " + probe.getCompactionThroughput() + " MiB/s");
+ double throughput = probe.getCompactionThroughputMebibytesAsDouble();
+
+ if (compactionThroughputAsDouble)
+ probe.output().out.println("Current compaction throughput: " + throughput + " MiB/s");
+ else
+ {
+ if (!DoubleMath.isMathematicalInteger(throughput))
+ throw new RuntimeException("You should use -d to get exact throughput in MiB/s");
Review Comment:
```suggestion
throw new RuntimeException("Use the -d flag to quiet this error and get the exact throughput in MiB/s");
```
##########
test/unit/org/apache/cassandra/tools/nodetool/SetGetEntireSSTableStreamThroughputTest.java:
##########
@@ -34,6 +34,8 @@
*/
public class SetGetEntireSSTableStreamThroughputTest extends CQLTester
{
+ private static final int MAX_INT_CONFIG_VALUE_MIB = Integer.MAX_VALUE - 1;
Review Comment:
ah ok, max value is `Integer.MAX_VALUE - 1`
##########
src/java/org/apache/cassandra/tools/nodetool/GetCompactionThroughput.java:
##########
@@ -17,17 +17,34 @@
*/
package org.apache.cassandra.tools.nodetool;
+import com.google.common.math.DoubleMath;
+
import io.airlift.airline.Command;
+import io.airlift.airline.Option;
import org.apache.cassandra.tools.NodeProbe;
import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
-@Command(name = "getcompactionthroughput", description = "Print the MiB/s throughput cap for compaction in the system")
+@Command(name = "getcompactionthroughput", description = "Print the MiB/s throughput cap for compaction in the system as a rounded number")
public class GetCompactionThroughput extends NodeToolCmd
{
+ @SuppressWarnings("UnusedDeclaration")
+ @Option(name = { "-d", "--precise-mib" }, description = "Print the MiB/s throughput cap for compaction in the system as a precise number (double)")
+ private boolean compactionThroughputAsDouble;
+
@Override
public void execute(NodeProbe probe)
{
- probe.output().out.println("Current compaction throughput: " + probe.getCompactionThroughput() + " MiB/s");
+ double throughput = probe.getCompactionThroughputMebibytesAsDouble();
+
+ if (compactionThroughputAsDouble)
+ probe.output().out.println("Current compaction throughput: " + throughput + " MiB/s");
+ else
+ {
+ if (!DoubleMath.isMathematicalInteger(throughput))
+ throw new RuntimeException("You should use -d to get exact throughput in MiB/s");
+
+ probe.output().out.println("Current compaction throughput: " + probe.getCompactionThroughput() + " MB/s");
Review Comment:
good point, when will we switch to MiB/s?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org