You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2022/09/29 07:26:50 UTC
[GitHub] [cassandra] bereng opened a new pull request, #1891: Cassandra 14227 trunk
bereng opened a new pull request, #1891:
URL: https://github.com/apache/cassandra/pull/1891
Thanks for sending a pull request! Here are some tips if you're new here:
* Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
* Be sure to keep the PR description updated to reflect all changes.
* Write your PR title to summarize what this PR proposes.
* If possible, provide a concise example to reproduce the issue for a faster review.
* Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
* If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
Commit messages should follow the following format:
```
<One sentence description, usually Jira title or CHANGES.txt summary>
<Optional lengthier description (context on patch)>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1205042939
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
No I didn't mean to remove the auto detection at all. Just to have a single ant_repeat job but that doesn't allow to run multiple repeats in a single push as you say...
One problem I hit, iiuc, is that when specifying a test to repeat the auto detection would still kick in and add lots of tests I didn't want to repeat at the time. I thought it would be useful to add a flag to skip auto detection so I have a way to repeat only what I want. It does also let me call generate without having to have extra checked out branches when doing quick tests in aux repos i.e. I added -s [wdyt](https://github.com/bereng/cassandra/commit/03842aa1b8dfd6149f08dbfdb52c3eeabe5d9330)?
As per the job renames and repeats here is an [example](https://app.circleci.com/pipelines/github/bereng/cassandra/946/workflows/baf437ce-7198-4c4c-9530-9d59b42760f8) on my drafts branch. I think all is ready now and we only need to wait for sstable tries to merge and rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1196804520
##########
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeTTLOverflowUpgradeTest.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.upgrade;
+
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.Attributes;
+import org.apache.cassandra.db.rows.Cell;
+import org.apache.cassandra.distributed.api.ICoordinator;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.StorageCompatibilityMode;
+import org.assertj.core.api.Assertions;
+
+import static org.apache.cassandra.distributed.api.ConsistencyLevel.ALL;
+import static org.apache.cassandra.utils.StorageCompatibilityMode.NONE;
+import static org.apache.cassandra.utils.StorageCompatibilityMode.UPGRADING;
+
+/**
+ * Tests TTL the overflow policy triggers at the correct limit: year 2038 <=nc or 2186 >nc
+ * <p>
+ * <=nc overflow policy triggers at year 2038. That could be <=4.1 or 5.0 with 4.x storage compatibility
+ * >nc overflow policy triggers at year 2106. That is >=5.0 using >=5.x storage compatibility
+ *
+ * @see StorageCompatibilityMode
+ */
+public class MixedModeTTLOverflowUpgradeTest extends UpgradeTestBase
+{
+ @Test
+ public void testTTLOverflowDuringUpgrade() throws Throwable
+ {
+ testTTLOverflow((cluster, node) -> {
+ if (node == 1) // only node1 is upgraded, and the cluster is in mixed versions mode
+ {
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+
+ // We restart the upgraded node out of 4.0 storage compatibility,
+ // and we set it to be compatible with 5.0.
+ // 2038 should still be the limit because there is still a not upgraded node.
+ cluster.get(1).shutdown().get();
+ cluster.get(1).config().set("storage_compatibility_mode", UPGRADING.toString());
+ cluster.get(1).startup();
Review Comment:
Nit: We can wrap these block of three lines into a utility method, this way: https://github.com/adelapena/cassandra/commit/5faf80a8c137f1c7a9f2d26ab65fdcd31a2e50c8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210274977
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
+ * But many tests don't load the config hence we need to force it otherwise they would run always under the default.
+ *
+ * This is a dev/CI only property. Do not use otherwise.
Review Comment:
Aaah, right, it's on the above paragraph! Yes, that's exactly what I meant. Sorry I've missed it, I left the comment on the second paragraph and didn't see the change on the first one after the rebase. +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122628399
##########
src/java/org/apache/cassandra/db/ExpirationDateOverflowHandling.java:
##########
@@ -62,23 +62,23 @@
}
public static final String MAXIMUM_EXPIRATION_DATE_EXCEEDED_WARNING = "Request on table {}.{} with {}ttl of {} seconds exceeds maximum supported expiration " +
- "date of 2038-01-19T03:14:06+00:00 and will have its expiration capped to that date. " +
+ "date of 2106-02-07T06:28:14+00:00 and will have its expiration capped to that date. " +
Review Comment:
All these batched in the original comment on this subject
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122645664
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,8 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
Review Comment:
Right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122636063
##########
src/java/org/apache/cassandra/io/sstable/IndexInfo.java:
##########
@@ -47,7 +47,7 @@
* (long) IndexInfo.offset
* (long) IndexInfo.width
* (bool) IndexInfo.endOpenMarker != null (if 3.0)
- * (int) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)
+ * (long) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)
Review Comment:
I changed it to `(int) (Uint post c14427) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)` wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1449689989
Yes, I have an option to resolve (and unresolve) comments here. If commenters are unable to resolve their own comments that’s a permission issue we should resolve with ASF Infra.
We don’t have well defined etiquettes here, but the author should not just assume it’s fine because they want to - I know several reviewers whose process would be disrupted by this, and creating additional burden for reviewers is not cool. You should at least ask reviewers if they are ok with this before doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1118622237
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +45,14 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ // -5 to leave room for any future sentinel values just in case
Review Comment:
In the latest commit i move it to -2. Future sentinels could be values in the past. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1449645742
As usual if I resolved some conversation prematurely let me know and I'll reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129237812
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -88,15 +91,15 @@ public StreamingTombstoneHistogramBuilder(int maxBinSize, int maxSpoolSize, int
*
* @param point the point to be added
*/
- public void update(int point)
+ public void update(long point)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127825407
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
I'm not sure adding the boolean array doesn't make sense since it would still save some memory and maybe conversions. But it might well not be a noticeable impact so I'm not against leaving it as it was.
As for resolving, reviewers cannot resolve conversations, just add a comment indicating agreement so the assignee can close them. I have left such comments on most conversations so you can close them.
However I'd keep this particular conversation open for a while, since there are other reviewers and they might find it interesting. With such a large PR it's easy to miss things so I think it's good to see what has caught the attention of other reviewers. I hope it's not too distracting to leave a few conversations open, specially if you use email notifications for tracking discussions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129033954
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
I think the comment you're asking for is there [already](https://github.com/apache/cassandra/pull/1891/files#diff-dbf9d67e1f6060176920b0d2555d111ef2d0fc0c3c3a0bec8e49c907155194eeR331)?
The 'extra' rows you are seeing are being inserted by the '[prepare](https://github.com/apache/cassandra/pull/1891/files#diff-dbf9d67e1f6060176920b0d2555d111ef2d0fc0c3c3a0bec8e49c907155194eeR394)()' methods iiuc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1104862940
##########
src/java/org/apache/cassandra/db/rows/ArrayCell.java:
##########
@@ -34,19 +36,20 @@ public class ArrayCell extends AbstractCell<byte[]>
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new ArrayCell(ColumnMetadata.regularColumn("", "", "", ByteType.instance), 0L, 0, 0, EMPTY_BYTE_ARRAY, null));
+ // Careful: Adding vars here has an impact on memtable size
private final long timestamp;
private final int ttl;
- private final int localDeletionTime;
+ private final int localDeletionTimeGuavaUint;
private final byte[] value;
private final CellPath path;
- public ArrayCell(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, byte[] value, CellPath path)
+ public ArrayCell(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, byte[] value, CellPath path)
{
super(column);
this.timestamp = timestamp;
this.ttl = ttl;
- this.localDeletionTime = localDeletionTime;
+ this.localDeletionTimeGuavaUint = Cell.deletionTimeLongToGuavaUint(localDeletionTime);
Review Comment:
```suggestion
this.localDeletionTimeGuavaUint = deletionTimeLongToGuavaUint(localDeletionTime);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108288370
##########
test/unit/org/apache/cassandra/cql3/validation/entities/SecondaryIndexTest.java:
##########
@@ -1575,7 +1575,7 @@ private void testIndexOnPartitionKeyInsertExpiringColumn(boolean flushBeforeUpda
flush();
execute("UPDATE %s USING TTL 1 SET b = 10 WHERE k1 = 1 AND k2 = 2");
- Thread.sleep(1000);
+ Thread.sleep(1500);
Review Comment:
What would be a good alternative?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109537064
##########
test/unit/org/apache/cassandra/utils/CassandraGenerators.java:
##########
@@ -99,7 +100,7 @@
private static final Gen<SinglePartitionReadCommand> SINGLE_PARTITION_READ_COMMAND_GEN = gen(rnd -> {
TableMetadata metadata = TABLE_METADATA_GEN.generate(rnd);
- int nowInSec = (int) rnd.next(Constraint.between(1, Integer.MAX_VALUE));
+ long nowInSec = rnd.next(Constraint.between(1, Cell.MAX_DELETION_TIME - 1));
Review Comment:
Probably a left over from some previous sentinel value collision. Removed and confirmed with CI all is good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107085531
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -133,8 +133,11 @@ static class BigVersion extends Version
// na (4.0-rc1): uncompressed chunks, pending repair session, isTransient, checksummed sstable metadata file, new Bloomfilter format
// nb (4.0.0): originating host id
+ // nc (5.0): Long deletionTime to prevent TTL overflow
Review Comment:
Isn't it `oa`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105731270
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
It really shouldn't. The cast is free, you just load into a suitably sized register. The truncation will have a single cycle latency (and greater than one-per-cycle throughput). If you can measure the impact of this in a real C* trace, I will eat my hat. If you can even get it to show up in a microbenchmark that does _anything_ real (compaction, memtable read or anything really) I will be very impressed. It's going to be 100% swallowed up by memory latency, data dependencies in the pipeline, etc. You only worry about this kind of thing if you can get far better instructions-per-cycle than we can. This is the lowest of low level optimisation concern.
However, the wasted memory will definitely have an impact. For many JVMs this will require an extra 8-bytes of memory.
So this is a pretty simple tradeoff, and I would not personally waste my time benchmarking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107007919
##########
test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java:
##########
@@ -130,8 +130,8 @@ public void testAggressiveFullyExpired()
Util.flush(cfs);
Set<SSTableReader> sstables = Sets.newHashSet(cfs.getLiveSSTables());
- int now = (int)(System.currentTimeMillis() / 1000);
- int gcBefore = now + 2;
+ long now = System.currentTimeMillis() / 1000;
Review Comment:
Nit: This could be a call to `FBUtilities.nowInSeconds()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106752734
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
Testing this the problem is that when we're scrubbing legacy sstables with overflowed ldts we need to support reading negative ldts. Moving this to an uint is not possible then which is why probably this wasn't moved in the first place. I leave this one pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106756031
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
Extend DeletionTime for those cases, so that it interprets as signed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106806624
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
So you're talking about the reverse conversion. The annotated code is not protected at all, and so there is no reason to box the value here. But, yes, protection in the reverse direction can be easily reproduced with a static method for converting from long to int, that confirms the upper bits are unset, i.e. simply checks (value & 0xffffffff00000000L) == 0.
There is no reason to use a boxing method for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1173386177
##########
conf/jvm-server.options:
##########
@@ -181,8 +181,8 @@
###################################
# Defines how to handle INSERT requests with TTL exceeding the maximum supported expiration date:
-# * REJECT: this is the default policy and will reject any requests with expiration date timestamp after 2038-01-19T03:14:06+00:00.
-# * CAP: any insert with TTL expiring after 2038-01-19T03:14:06+00:00 will expire on 2038-01-19T03:14:06+00:00 and the client will receive a warning.
+# * REJECT: this is the default policy and will reject any requests with expiration date timestamp after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 depending on compatibility mode
+# * CAP: any insert with TTL expiring after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 will expire on 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 and the client will receive a warning.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1173381705
##########
test/unit/org/apache/cassandra/cql3/CustomNowInSecondsTest.java:
##########
@@ -73,6 +79,48 @@ private void testSelectQuery(boolean prepared)
assertEquals(0, executeSelect(format("SELECT * FROM %s.%s", ks, tbl), now + day + 1, prepared).size());
}
+ @Test
+ public void testSelectQueryWitInthOverflow()
+ {
+ testSelectQueryOverflowingIntTimestamps(false);
+ testSelectQueryOverflowingIntTimestamps(true);
+ }
+
+ private void testSelectQueryOverflowingIntTimestamps(boolean prepared)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1184933779
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -69,11 +73,19 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
{
private final static Logger logger = LoggerFactory.getLogger(BigFormat.class);
+ // Feature flag env variable to use extended TTL up to 2106 with the new oa sstable format. See c14227
+ public static enum TTL_MODE {LEGACY_TTL_2038, COMPATIBILITY, EXTENDED_TTL_2106};
+ @VisibleForTesting
+ public static final String USE_OA_SSTABLE_FORMAT = "cassandra.use_oa_sstable_format";
Review Comment:
What about `cassandra.sstable.format.ttl_mode`? We already have that prefix on `cassandra.sstable.format.default`, in `CassandraRelevantProperties.SSTABLE_FORMAT_DEFAULT`, added by [CEP-17](https://issues.apache.org/jira/browse/CASSANDRA-17056).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1535831029
Just pushed the rename of the FF. Also please let me know if I can close all the open comments we have. Thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185996004
##########
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeTTLOverflowUpgradeTest.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.upgrade;
+
+import org.junit.Test;
+
+import org.apache.cassandra.config.CassandraRelevantProperties;
+import org.apache.cassandra.cql3.Attributes;
+import org.apache.cassandra.db.rows.Cell;
+import org.apache.cassandra.distributed.api.ICoordinator;
+import org.apache.cassandra.io.sstable.format.big.BigFormat;
+import org.apache.cassandra.utils.Clock;
+import org.assertj.core.api.Assertions;
+
+import static org.apache.cassandra.distributed.api.ConsistencyLevel.ALL;
+
+/**
+ * Tests TTL the overflow policy triggers at the correct limit: year 2038 <=nc or 2186 >nc
+ *
+ * <=nc overflow policy triggers at year 2038. That could be <=4.1 or 5.0 with nc sstable format
+ * >nc overflow policy triggers at year 2106. That is >=5.0 and using >=oa sstable format
+ * See BigFormat.USE_OA_SSTABLE_FORMAT
+ */
+public class MixedModeTTLOverflowUpgradeTest extends UpgradeTestBase
+{
+ @Test
+ public void testTTLOverflowDuringUpgrade() throws Throwable
+ {
+ new TestCase()
+ .nodes(2)
+ .nodesToUpgradeOrdered(1, 2)
+ .upgradesToCurrentFrom(v41)
+ .setup(cluster -> {
+ cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, v int)"));
+
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+ })
+ .runAfterNodeUpgrade((cluster, node) -> {
+ if (node == 1) // only node1 is upgraded, and the cluster is in mixed versions mode
+ {
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
Review Comment:
We are not testing the scenario where there is a node in 5.0 compatible with 4.0 and another node still in 4.0. I have added that case in the commit changing the system property to `cassandra.storage_compatibility_version` that I mentioned above, https://github.com/adelapena/cassandra/commit/0a87f1ee13e5eed92dee637638667a30695c3426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185716815
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,8 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+ USE_OA_SSTABLE_FORMAT("cassandra.use_oa_sstable_format", BigFormat.TTL_MODE.LEGACY_TTL_2038.toString())
Review Comment:
Done
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -69,6 +71,10 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
{
private final static Logger logger = LoggerFactory.getLogger(BigFormat.class);
+ // Feature flag env variable to use extended TTL up to 2106 with the new oa sstable format. See c14227
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1132005703
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -242,47 +249,51 @@ void mergeNearestPoints()
{
assert isFull() : "DataHolder must be full in order to merge two points";
- final int[] smallestDifference = findPointPairWithSmallestDistance();
+ final long[] smallestDifference = findPointPairWithSmallestDistance();
- final int point1 = smallestDifference[0];
- final int point2 = smallestDifference[1];
+ final long point1 = smallestDifference[0];
+ final long point2 = smallestDifference[1];
- long key = wrap(point1, 0);
- int index = Arrays.binarySearch(data, key);
+ int index = Arrays.binarySearch(points, point1);
if (index < 0)
{
index = -index - 1;
- assert (index < data.length) : "Not found in array";
- assert (unwrapPoint(data[index]) == point1) : "Not found in array";
+ assert (index < points.length) : "Not found in array";
+ assert (points[index] == point1) : "Not found in array";
}
- long value1 = unwrapValue(data[index]);
- long value2 = unwrapValue(data[index + 1]);
-
- assert (unwrapPoint(data[index + 1]) == point2) : "point2 should follow point1";
+ long value1 = values[index];
+ long value2 = values[index + 1];
- long sum = value1 + value2;
+ assert (points[index + 1] == point2) : "point2 should follow point1";
- //let's evaluate in long values to handle overflow in multiplication
- int newPoint = saturatingCastToInt((point1 * value1 + point2 * value2) / sum);
+ long a = saturatingCastToLong(point1 * value1);
+ long b = saturatingCastToLong(point2 * value2);
+ long sum = saturatingCastToLong(value1 + value2);
+ long newPoint = saturatingCastToMaxDeletionTime(saturatingCastToLong(a + b) / sum);
+ newPoint = newPoint <= point1 ? saturatingCastToMaxDeletionTime(point1 + 1) : newPoint;
+ newPoint = newPoint > point2 ? point2 : newPoint;
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1469738569
The changes look good to me, I think I don't have anything else to add.
Maybe it's a good time to squash+rebase and see how CI looks like. There are some references to 4.2 that should be updated to 5.0 after the rebase.
I don't know what's the plan for attending @belliottsmith ask of preventing the major version dump. Are we going to do it here or in a separate ticket?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134126275
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,26 +60,31 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private int[] delTimesUnsignedIntegers;
private long boundaryHeapSize;
private int size;
- private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, int[] delTimes, long boundaryHeapSize, int size)
+ private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, long[] delTimes, long boundaryHeapSize, int size)
Review Comment:
We can just remove this private constructor, since we only call it from the `RangeTombstoneList(ClusteringComparator, int)` constructor to build the class with an empty array. That also makes `CassandraUInt.fromLong(long[])` unneeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135031506
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -146,11 +153,16 @@ public void add(RangeTombstone tombstone)
* This method will be faster if the new tombstone sort after all the currently existing ones (this is a common use case),
* but it doesn't assume it.
*/
- public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, int delTime)
+ public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, long delTime)
Review Comment:
Yeah but it's always been public and I don't want to burden any future callers with the uint conversion burden. I would prefer to leave it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1573458818
@adelapena I had to rebase bc of the sstable tries. I had to basically make sure the sstable version was getting to the places the DeletionTime serializer was needing it:
- J8 [here](https://app.circleci.com/pipelines/github/bereng/cassandra/970/workflows/2e949ede-cae5-4ab8-a118-43ec6d722b0d). All good as failures are C18065, c18499 and a timeout that has passed many times before in my tests
- J11 [green](https://app.circleci.com/pipelines/github/bereng/cassandra/970/workflows/b4fa699c-3f6b-4856-9357-67149a560257)
- J11/J17 as [expected](https://app.circleci.com/pipelines/github/bereng/cassandra/971/workflows/1f607fe6-d00d-4609-83e1-48c64ddd3029) J11 is green and j17 with the known failures
- J17/J17 with the [expected](https://app.circleci.com/pipelines/github/bereng/cassandra/971/workflows/ea8da6f9-ed4a-49fc-8bf2-d4a95e3f91a4) failures
Once the commit for the BtiFormat rebase is +1 I think we can merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by GitBox <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1290294127
Rebased with tries and tests [here](https://app.circleci.com/pipelines/github/bereng/cassandra). The only failure seems like an unrelated flaky I will look into which passes locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1104840216
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -282,6 +301,9 @@ public <V> Cell<V> deserialize(DataInputPlus in, LivenessInfo rowLiveness, Colum
}
}
+ if (localDeletionTime < 0)
+ localDeletionTime = helper.version < MessagingService.VERSION_42 ? LEGACY_OVERFLOWED_DELETION_TIME : Cell.deletionTimeGuavaUintToLong((int) localDeletionTime);
Review Comment:
Nit: break the long line
```suggestion
localDeletionTime = helper.version < MessagingService.VERSION_42
? LEGACY_OVERFLOWED_DELETION_TIME
: Cell.deletionTimeGuavaUintToLong((int) localDeletionTime);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1104865765
##########
src/java/org/apache/cassandra/db/rows/BufferCell.java:
##########
@@ -116,15 +117,15 @@ public CellPath path()
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
- return new BufferCell(newColumn, timestamp, ttl, localDeletionTime, value, path);
+ return new BufferCell(newColumn, timestamp, ttl, Cell.deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
Review Comment:
```suggestion
return new BufferCell(newColumn, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105704918
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,10 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
+ // Guava uint counterpart to the above for serialization purposes. This imposes a year 2186 max value
+ private static final int UNSET_NOWINSEC_GUAVA_UINT = UnsignedInteger.MAX_VALUE.intValue() - 1;
Review Comment:
I think this might be a copy-paste case form the `LEGACY_OVERFLOWED_DELETION_TIME` elsewhere. I'll change and run CI to make sure it so indeed as this is quite a self-contained case, I don't see any other interfering values getting in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106784459
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
What is it you think you’re gaining with this boxing? I have looked, and I cannot see any additional protection, and we have to rely on escape analysis to avoid significant performance penalty.
This kind of bit twiddling is all over the codebase and we can manage it very successfully. It *shouldn’t* be seen as exotic or dangerous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106997347
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -598,10 +598,12 @@ private static void loadLegacyTable(String tablePattern, String legacyVersion) t
* <p>
* Run this test alone (e.g. from your IDE) when a new version is introduced or format changed
* during development. I.e. remove the {@code @Ignore} annotation temporarily.
+ *
+ * *************RUN THIS TEST ALONE to avoid unwanted compactions, flushes, etc to interfere
Review Comment:
Does this mean that it should be moved to a separate class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107081377
##########
src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java:
##########
@@ -267,7 +271,24 @@ public int serializedSize(Version version, StatsMetadata component) throws IOExc
size += EstimatedHistogram.serializer.serializedSize(component.estimatedPartitionSize);
size += EstimatedHistogram.serializer.serializedSize(component.estimatedCellPerPartitionCount);
size += CommitLogPosition.serializer.serializedSize(component.commitLogIntervals.upperBound().orElse(CommitLogPosition.NONE));
- size += 8 + 8 + 4 + 4 + 4 + 4 + 8 + 8; // mix/max timestamp(long), min/maxLocalDeletionTime(int), min/max TTL, compressionRatio(double), repairedAt (long)
+ size += 8 + 8; // mix/max timestamp(long)
+ if (version.hasLongLocalDeletionTime())
Review Comment:
We have two identical if-else blocks, maybe we can merge them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107089769
##########
src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java:
##########
@@ -57,7 +57,8 @@ public static SSTableIdentityIterator create(SSTableReader sstable, RandomAccess
{
try
{
- DeletionTime partitionLevelDeletion = DeletionTime.serializer.deserialize(file);
+ DeletionTime partitionLevelDeletion;
+ partitionLevelDeletion = DeletionTime.getSerializer(sstable.descriptor.version).deserialize(file);
Review Comment:
```suggestion
DeletionTime partitionLevelDeletion = DeletionTime.getSerializer(sstable.descriptor.version).deserialize(file);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108183123
##########
test/unit/org/apache/cassandra/db/NativeCellTest.java:
##########
@@ -125,7 +125,7 @@ private static Cell<?> rndcell(ColumnMetadata col)
{
long timestamp = rand.nextLong();
int ttl = rand.nextInt();
- int localDeletionTime = rand.nextInt();
+ long localDeletionTime = Math.abs(rand.nextInt());
Review Comment:
Mmmmm it's not a uint we're generating... I think it's confusing calling `deletionTimeGuavaUintToLong()` on it. I'd rather leave it as it is. I'll resolve but feel free to slap me in the hand and reopen if you disagree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109568816
##########
test/unit/org/apache/cassandra/db/CellTest.java:
##########
@@ -416,20 +416,32 @@ private Cell<?> regular(TableMetadata cfm, String columnName, String value, long
return BufferCell.live(cdef, timestamp, ByteBufferUtil.bytes(value));
}
- private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, int localExpirationTime)
+ private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, long localExpirationTime)
{
return expiring(cfm, columnName, value, timestamp, 1, localExpirationTime);
}
- private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, int ttl, int localExpirationTime)
+ private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, int ttl, long localExpirationTime)
{
ColumnMetadata cdef = cfm.getColumn(ByteBufferUtil.bytes(columnName));
return new BufferCell(cdef, timestamp, ttl, localExpirationTime, ByteBufferUtil.bytes(value), null);
}
- private Cell<?> deleted(TableMetadata cfm, String columnName, int localDeletionTime, long timestamp)
+ private Cell<?> deleted(TableMetadata cfm, String columnName, long localDeletionTime, long timestamp)
{
ColumnMetadata cdef = cfm.getColumn(ByteBufferUtil.bytes(columnName));
return BufferCell.tombstone(cdef, timestamp, localDeletionTime);
}
+
+ private static void assertThrowsOnInvalidDeletionTime(CheckedFunction runnable)
Review Comment:
That is what I was looking for and I didn't manage to find originally. Thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108211756
##########
src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java:
##########
@@ -267,7 +271,24 @@ public int serializedSize(Version version, StatsMetadata component) throws IOExc
size += EstimatedHistogram.serializer.serializedSize(component.estimatedPartitionSize);
size += EstimatedHistogram.serializer.serializedSize(component.estimatedCellPerPartitionCount);
size += CommitLogPosition.serializer.serializedSize(component.commitLogIntervals.upperBound().orElse(CommitLogPosition.NONE));
- size += 8 + 8 + 4 + 4 + 4 + 4 + 8 + 8; // mix/max timestamp(long), min/maxLocalDeletionTime(int), min/max TTL, compressionRatio(double), repairedAt (long)
+ size += 8 + 8; // mix/max timestamp(long)
+ if (version.hasLongLocalDeletionTime())
Review Comment:
Imo you want to keep like this so it mirrors de/serialize() methods workflow making it easier to reason about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108340945
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,10 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
+ // Guava uint counterpart to the above for serialization purposes. This imposes a year 2186 max value
+ private static final int UNSET_NOWINSEC_GUAVA_UINT = UnsignedInteger.MAX_VALUE.intValue() - 1;
Review Comment:
MAX and MIN are often used in the code as sentinel values. You don't want sbdy passing you `UnsignedInteger.MAX_VALUE` to signal some special condition and accidentally mapping that to `UNSET_NOWINSEC`. So the -1 is to avoid that collision.
`MAX_VALUE-1` imo is a 'better' sentinel value, if such a thing can be said about any random value, because it's far less likely to collide and allows consumers to use the 'classic' MAX sentinel values they are used to.
But no other hard reasons as nowInSec won't be around MAX values until year 2106.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1480666588
Thanks for the clarification guys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1126369669
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
Just took a look at the last commit storing uint, which is very similar to what I was thinking. However, I think it misses the case where we add a range tombstone with a `InvalidDeletionTime`.
On on hand, the valid-invalid state of an invalid deletion time is codified by the implemented class, so we can use a int for codifying the local deletion time. On the other hand, `RangeTombstoneList` seems to be an attempt at saving memory by aggregating multiple range tombstones, so we have an array of local deletion times. If this array is comprised by uints instead of longs, we miss a representation of the valid/invalid status that we used to represent with the class.
I think this tiny test demonstrates it:
```java
// create an invalid deletion time
DeletionTime dt = DeletionTime.build(1, -1);
assertFalse(dt.validate());
// use the invalid deletion time for a range tombstone and aggregate it
RangeTombstoneList rtl = new RangeTombstoneList(null, 1);
rtl.add(new RangeTombstone(Slice.ALL, dt));
// undo the aggregation and see if the deletion time is still invalid
dt = rtl.iterator().next().deletionTime();
assertNotNull(dt);
assertFalse(dt.validate()); // fails
```
I guess we could try to overcome this problem by also storing an array of booleans on `RangeTombstoneList` storing what deletion times are invalid. Still, I'm not sure whether it's worth it and I wouldn't oppose to leaving `RangeTombstoneList` with a `long` array. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1126464233
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
I was waiting for CI to finish... But yeah if you add booleans then it doesn't make sense... :thinking: ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121744023
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
Makes sense to me, that would be used by the code that has been commented in `CompactionsCQLTest`. What are the other changes that we need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122134740
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,8 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
Review Comment:
I think this can be `Long.MIN_VALUE` for better clarity, since this value is compared with longs, it doesn't seem to leave the class and it doesn't participate in serialization.
Keeping it as an int could make readers think that choosing `Integer.MIN_VALUE` over `Long.MIN_VALUE` has anything to do with unsigned ints, which I think isn't the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122918836
##########
NEWS.txt:
##########
@@ -1,9 +1,9 @@
-PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092)
-------------------------------------------------------------------
+PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092 & CASSANDRA-14227)
+-----------------------------------------------------------------------------------
(General upgrading instructions are available in the next section)
-The maximum expiration timestamp that can be represented by the storage engine is
-2038-01-19T03:14:06+00:00, which means that inserts with TTL thatl expire after
+The maximum expiration timestamp that can be represented by the storage engine has been
+raised to 2106-02-07T06:28:14+00:00, which means that inserts with TTL that expire after
Review Comment:
Oh, right, makes sense, +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1449778160
Oh icwym now. When I am reviewing I use the trail of emails I get that have links direct to particular comments so that's not a problem for me. But if you're a reviewer coming straight to a PR you just see a sea of resolved comments...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121677961
##########
src/java/org/apache/cassandra/utils/CassandraUInt.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.utils;
+
+import com.google.common.primitives.UnsignedInteger;
+import com.google.common.primitives.UnsignedInts;
+
+/**
+ * This class packages several UInt utility methods so we can easily change it's implementation when needed.
+ */
+public class CassandraUInt
+{
+
+ private static long MAX_VALUE_LONG = UnsignedInteger.MAX_VALUE.longValue();
+ private static int MAX_VALUE_UINT = UnsignedInteger.MAX_VALUE.intValue();
Review Comment:
These two can be `final`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1119906485
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
Right you are, icym now. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122636063
##########
src/java/org/apache/cassandra/io/sstable/IndexInfo.java:
##########
@@ -47,7 +47,7 @@
* (long) IndexInfo.offset
* (long) IndexInfo.width
* (bool) IndexInfo.endOpenMarker != null (if 3.0)
- * (int) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)
+ * (long) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)
Review Comment:
I changed it to `(int (uint post c14427))` wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122184147
##########
src/java/org/apache/cassandra/db/ReadCommand.java:
##########
@@ -1097,7 +1100,7 @@ public long serializedSize(ReadCommand command, int version)
return 2 // kind + flags
+ (command.isDigestQuery() ? TypeSizes.sizeofUnsignedVInt(command.digestVersion()) : 0)
+ command.metadata().id.serializedSize()
- + TypeSizes.sizeof(command.nowInSec())
+ + TypeSizes.INT_SIZE
Review Comment:
```suggestion
+ TypeSizes.INT_SIZE // command.nowInSec() is serialized as uint
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121940748
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +45,14 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ // -5 to leave room for any future sentinel values just in case
Review Comment:
So MAX - 2 is max deletion time, MAX - 1 is future use, and MAX is no deletion, right? If we keep a reserved future sentinel we should probably keep the previous comment, `-2 to leave room for any future sentinel values just in case`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127940030
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
I think that's all. It was my idea to provide that code you can uncomment ready for the next dev that might need it. Everything else needed I baked seamlessly into the existing code already, such as having KS as a param here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185913169
##########
test/unit/org/apache/cassandra/cql3/EmptyValuesTest.java:
##########
@@ -76,6 +78,11 @@ private void verify(String emptyValue) throws Throwable
try (PrintStream out = new PrintStream(buf, true))
{
ProcessBuilder pb = new ProcessBuilder("tools/bin/sstabledump", ssTable.getFilename());
+ Map<String, String> envs = pb.environment();
+ envs.put("JVM_OPTS", "-D"
+ + CassandraRelevantProperties.SSTABLE_FORMAT_TTL_MODE.getKey()
+ + "="
+ + CassandraRelevantProperties.SSTABLE_FORMAT_TTL_MODE.getString());
Review Comment:
What's the purpose of this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1187202017
##########
test/unit/org/apache/cassandra/cql3/EmptyValuesTest.java:
##########
@@ -76,6 +78,11 @@ private void verify(String emptyValue) throws Throwable
try (PrintStream out = new PrintStream(buf, true))
{
ProcessBuilder pb = new ProcessBuilder("tools/bin/sstabledump", ssTable.getFilename());
+ Map<String, String> envs = pb.environment();
+ envs.put("JVM_OPTS", "-D"
+ + CassandraRelevantProperties.SSTABLE_FORMAT_TTL_MODE.getKey()
+ + "="
+ + CassandraRelevantProperties.SSTABLE_FORMAT_TTL_MODE.getString());
Review Comment:
I see in your latest proposed commit you removed that. The test will fail if you do so with `test-oa`, it's needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1187194772
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,8 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+ USE_OA_SSTABLE_FORMAT("cassandra.use_oa_sstable_format", BigFormat.TTL_MODE.LEGACY_TTL_2038.toString())
Review Comment:
Solved differently in later commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1205338885
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
I'll create a ticket for the -s.
And yes, after the rebase, I will manually choose all tests that have changes that are more than a simple int/long toggle. Then I'll batch them up for repeated testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210167679
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
+ * But many tests don't load the config hence we need to force it otherwise they would run always under the default.
+ *
+ * This is a dev/CI only property. Do not use otherwise.
Review Comment:
I don't see any changes here, maybe you forgot to merge/push?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210389237
##########
conf/cassandra.yaml:
##########
@@ -1956,3 +1956,30 @@ drop_compact_storage_enabled: false
# big:
# param1: value1
# ...
+
+# This property indicates with what Cassandra major version the storage format will be compatible with.
+#
+# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
+#
+# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
+# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,
+# so that feature won't be available if this property is set to CASSANDRA_4. See upgrade guides for details. Currently
+# the only supported major in CASSANDRA_4.
+#
+# Possible values are in the StorageCompatibilityMode.java file accessible online. At the time of writing these are:
+# - CASSANDRA_Y: Stays compatible with the Y.x line in features, formats and component versions
+# - UPGRADING: The cluster monitors nodes versions during this interim stage. _This has a cost_ but ensures any new features,
+# formats, versions, etc are enabled safely.
+# - NONE: Start with all the new features and formats enabled.
+#
+# A typical upgrade would be:
+# - Do a rolling upgrade starting all nodes in CASSANDRA_Y compatibility mode.
+# - Once the new binary is rendered stable do a rolling restart with UPGRADING. The cluster will enable new features in a safe way
+# until all nodes are started in UPGRADING, then all new features are enabled.
+# - Do a rolling restart with all nodes starting with NONE. This sheds the extra cost of checking nodes versions and ensures
+# a stable cluster. If a node from a previous version was started by accident we won't any longer toggle behaviors as when UPGRADING.
+
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134062355
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +255,31 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ private InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
Review Comment:
After the last changes `localDeletionTime` isn't used anymore, since we use the generic sentinel value. So we can remove the argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134126275
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,26 +60,31 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private int[] delTimesUnsignedIntegers;
private long boundaryHeapSize;
private int size;
- private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, int[] delTimes, long boundaryHeapSize, int size)
+ private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, long[] delTimes, long boundaryHeapSize, int size)
Review Comment:
We can just remove this private constructor, since we only call it from the `RangeTombstoneList(ClusteringComparator, int)` constructor to build the class with an empty array. That also makes `CassandraUInt.fromLong` unneeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135027691
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +255,31 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ private InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
Review Comment:
That I like a lot and it stresses the fact we shed the actual value and use a sentinel instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134112563
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -146,11 +153,16 @@ public void add(RangeTombstone tombstone)
* This method will be faster if the new tombstone sort after all the currently existing ones (this is a common use case),
* but it doesn't assume it.
*/
- public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, int delTime)
+ public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, long delTime)
+ {
+ add(start, end, markedAt, CassandraUInt.fromLong(delTime));
+ }
+
+ public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, int delTimeUnsignedInteger)
Review Comment:
Can be `private`, helping to hide the uint signatures to the rest of the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135043138
##########
src/java/org/apache/cassandra/db/compaction/Scrubber.java:
##########
@@ -614,7 +619,84 @@ public Unfiltered next()
}
nextToOffer = null;
- return next;
+ return computeFinalRow((Row) next);
+ }
+
+ private Row computeFinalRow(Row next)
+ {
+ // If the row has overflowed let rows skip them unless we need to keep them for the overflow policy
+ if (hasOverflowedLocalExpirationTimeRow((Row) next) && !reinsertOverflowedTTLRows)
+ return null;
+ else if (reinsertOverflowedTTLRows)
+ return rebuildTimestamptsForOverflowedRows(next);
+ else
+ return next;
+ }
+
+ /*
+ * When building ldt on deser it won't overflow now being a long as it used to. This causes row resurrection for old sstable formats!
+ * To prevent it we preserve the overflow to be backwards compatible and to feed into the overflow policy
+ */
+ private Row rebuildTimestamptsForOverflowedRows(Row row)
+ {
+ if (sstableVersion.hasLongLocalDeletionTime())
+ return row;
+
+ LivenessInfo livenessInfo = row.primaryKeyLivenessInfo();
+ if (livenessInfo.isExpiring() && livenessInfo.localExpirationTime() >= 0)
+ {
+ livenessInfo = livenessInfo.withUpdatedTimestampAndLocalDeletionTime(livenessInfo.timestamp(), livenessInfo.localExpirationTime(), false);
+ }
+
+ return row.transformAndFilter(livenessInfo, row.deletion(), cd -> {
+ if (cd.column().isSimple())
+ {
+ Cell cell = (Cell)cd;
+ return cell.isExpiring() && cell.localDeletionTime() >= 0
+ ? cell.withUpdatedTimestampAndLocalDeletionTime(cell.timestamp(), cell.localDeletionTime())
+ : cell;
+ }
+ else
+ {
+ ComplexColumnData complexData = (ComplexColumnData)cd;
+ return complexData.transformAndFilter(cell -> cell.isExpiring() && cell.localDeletionTime() >= 0
+ ? cell.withUpdatedTimestampAndLocalDeletionTime(cell.timestamp(), cell.localDeletionTime())
+ : cell);
+ }
+ }).clone(HeapCloner.instance);
+ }
+
+ private boolean hasOverflowedLocalExpirationTimeRow(Row next)
+ {
+ if (sstableVersion.hasLongLocalDeletionTime())
+ return false;
+
+ Row row = next;
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1136586192
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,26 +60,31 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private int[] delTimesUnsignedIntegers;
private long boundaryHeapSize;
private int size;
- private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, int[] delTimes, long boundaryHeapSize, int size)
+ private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, long[] delTimes, long boundaryHeapSize, int size)
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1196778384
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
+ * But many tests don't load the config hence we need to force it otherwise they would run always under the default.
+ *
+ * This is a dev/CI only property. Do not use otherwise.
Review Comment:
Nit: maybe we can repeat here the nice explanation that there is in `DatabaseDescriptor.getStorageCompatibilityMode`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1449653444
It’s usually preferable to let the reviewer resolve their own feedback if possible, to avoid that dance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1450122131
Sure we can do that but lets close the obvious immediate ones at least.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122181265
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
This class seems to be able to internally store uint deletion times to save some memory, as we do with `DeletionTime`. Not sure if it is worth it, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122632354
##########
src/java/org/apache/cassandra/db/ReadCommand.java:
##########
@@ -1097,7 +1100,7 @@ public long serializedSize(ReadCommand command, int version)
return 2 // kind + flags
+ (command.isDigestQuery() ? TypeSizes.sizeofUnsignedVInt(command.digestVersion()) : 0)
+ command.metadata().id.serializedSize()
- + TypeSizes.sizeof(command.nowInSec())
+ + TypeSizes.INT_SIZE
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108171619
##########
src/java/org/apache/cassandra/db/rows/BufferCell.java:
##########
@@ -34,21 +34,34 @@ public class BufferCell extends AbstractCell<ByteBuffer>
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new BufferCell(ColumnMetadata.regularColumn("", "", "", ByteType.instance), 0L, 0, 0, ByteBufferUtil.EMPTY_BYTE_BUFFER, null));
+ // Careful: Adding vars here has an impact on memtable size
private final long timestamp;
private final int ttl;
- private final int localDeletionTime;
+ private final int localDeletionTimeGuavaUint;
private final ByteBuffer value;
private final CellPath path;
- public BufferCell(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, ByteBuffer value, CellPath path)
+ public BufferCell(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, ByteBuffer value, CellPath path)
Review Comment:
Right, stupid mistake apologies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108176492
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -94,7 +94,7 @@
* See {@link #testGenerateSstables()} to generate sstables.
* Take care on commit as you need to add the sstable files using {@code git add -f}
*/
- public static final String[] legacyVersions = {"nb", "na", "me", "md", "mc", "mb", "ma"};
+ public static final String[] legacyVersions = {"oa", "nb", "na", "me", "md", "mc", "mb", "ma"};
Review Comment:
Yes but it needs to be added here. `When adding a new sstable version, add that one here` just a few lines above. Also notice how `nb` was there when it was the current version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1110067703
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -598,10 +598,12 @@ private static void loadLegacyTable(String tablePattern, String legacyVersion) t
* <p>
* Run this test alone (e.g. from your IDE) when a new version is introduced or format changed
* during development. I.e. remove the {@code @Ignore} annotation temporarily.
+ *
+ * *************RUN THIS TEST ALONE to avoid unwanted compactions, flushes, etc to interfere
Review Comment:
Not a big deal, but I think it's clearer to have the comment next to the `@Ignore` tag, since it explains that tag. The `TODO` prefix is highlighted and shown in code analyzers by some IDEs, which might be useful to remind us that at some point we have to fix the test (separate ticket, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1118623488
##########
test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java:
##########
@@ -313,16 +314,18 @@ public void testCompactionInvalidTombstone() throws Throwable
{
DatabaseDescriptor.setCorruptedTombstoneStrategy(Config.CorruptedTombstoneStrategy.exception);
prepare();
+
// write a standard tombstone with negative local deletion time (LDTs are not set by user and should not be negative):
- RowUpdateBuilder rub = new RowUpdateBuilder(getCurrentColumnFamilyStore().metadata(), -1, System.currentTimeMillis() * 1000, 22).clustering(33).delete("b");
- rub.build().apply();
- flush();
- compactAndValidate();
- readAndValidate(true);
- readAndValidate(false);
+ Assertions.assertThatThrownBy(() -> {
+ new RowUpdateBuilder(getCurrentColumnFamilyStore().metadata(),
+ -1,
+ System.currentTimeMillis() * 1000,
+ 22).clustering(33).delete("b");
+ }).isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("is outside the range for an unsigned integer value");
}
- @Test
+ @Test(expected = RuntimeException.class) // msg = "is outside the range for an unsigned integer value"
Review Comment:
In the latest commit I generated some corrupted sstables, load them and trigger the sstable corruption detection path. which is the original intent of the ticket afaiu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1451610408
> The exception to this are trivial suggestions that are addressed exactly as suggested, such adding an @Override annotation, fixing typos, and that kind of nits. IMO those conversations are unlikely to be discussed by the reviewer and thus they can be immediately resolved.
I know that some reviewers prefer to handle closing even these, as it is very easy for an author to make mistakes when closing them. I have been on both ends of this - having failed to handle something I thought I had handled and had closed, and having had resolved something that wasn't. So it can be helpful to have a blanket policy of leaving it to the reviewer, and trying to have close rounds of review. But, I agree this isn't _so_ important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122890854
##########
src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java:
##########
@@ -587,7 +588,10 @@ public Row deserializeRowBody(DataInputPlus in,
{
long timestamp = header.readTimestamp(in);
int ttl = hasTTL ? header.readTTL(in) : LivenessInfo.NO_TTL;
- int localDeletionTime = hasTTL ? header.readLocalDeletionTime(in) : LivenessInfo.NO_EXPIRATION_TIME;
+ long localDeletionTime = hasTTL ? header.readLocalDeletionTime(in) : LivenessInfo.NO_EXPIRATION_TIME;
+ if (localDeletionTime < 0)
+ localDeletionTime = helper.version < MessagingService.VERSION_42 ? Cell.LEGACY_OVERFLOWED_DELETION_TIME : Cell.deletionTimeUnsignedIntegerToLong((int) localDeletionTime);
Review Comment:
Great, thanks. Just a trivial nit, there is a missed space right before the `<` operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1123064166
##########
test/unit/org/apache/cassandra/utils/CassandraGenerators.java:
##########
@@ -99,7 +100,7 @@
private static final Gen<SinglePartitionReadCommand> SINGLE_PARTITION_READ_COMMAND_GEN = gen(rnd -> {
TableMetadata metadata = TABLE_METADATA_GEN.generate(rnd);
- int nowInSec = (int) rnd.next(Constraint.between(1, Integer.MAX_VALUE));
+ long nowInSec = rnd.next(Constraint.between(1, Cell.MAX_DELETION_TIME));
Review Comment:
Our codebase will now be limited to 2186. So you want to test/use values up to that rather than only up to `Integer.MAX_VALUE` as `nowInSec` is now a `long` instead of an `int`. Am I answering your question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134433668
##########
src/java/org/apache/cassandra/db/compaction/Scrubber.java:
##########
@@ -614,7 +619,84 @@ public Unfiltered next()
}
nextToOffer = null;
- return next;
+ return computeFinalRow((Row) next);
+ }
+
+ private Row computeFinalRow(Row next)
+ {
+ // If the row has overflowed let rows skip them unless we need to keep them for the overflow policy
+ if (hasOverflowedLocalExpirationTimeRow((Row) next) && !reinsertOverflowedTTLRows)
Review Comment:
Unneeded cast to `Row`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135329486
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,26 +60,31 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private int[] delTimesUnsignedIntegers;
private long boundaryHeapSize;
private int size;
- private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, int[] delTimes, long boundaryHeapSize, int size)
+ private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, long[] delTimes, long boundaryHeapSize, int size)
Review Comment:
Great, thanks. We could also break the long line, which has become even longer than it already was:
```java
private RangeTombstoneList(ClusteringComparator comparator,
ClusteringBound<?>[] starts,
ClusteringBound<?>[] ends,
long[] markedAts,
int[] delTimesUnsignedIntegers,
long boundaryHeapSize,
int size)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135345974
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +44,15 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ public static final int NO_DELETION_TIME_UNSIGNED_INTEGER = CassandraUInt.MAX_VALUE_UINT;
+ public static final long MAX_DELETION_TIME = CassandraUInt.MAX_VALUE_LONG - 2;
+ public static final int MAX_DELETION_TIME_UNSIGNED_INTEGER = CassandraUInt.fromLong(MAX_DELETION_TIME);
Review Comment:
Fine with me. I'm starting to think that not only `MAX_DELETION_TIME_UNSIGNED_INTEGER` but all the `*DELETION_TIME*` constants could be more naturally placed in `DeletionTime` than in `Cell`. But that's debatable, and in any case we don't have to do it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1133598531
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
The magic of the weekend thinking: Managed to make RTL uint based with no extra arrays or anything :-) We already have a sentinel Uint for invalid ldts so just use that... wdyt? see latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134074422
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,17 +42,41 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ protected final int localDeletionTimeUnsignedInteger;
Review Comment:
Nit: can have default package visibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1104864366
##########
src/java/org/apache/cassandra/db/rows/ArrayCell.java:
##########
@@ -83,23 +86,23 @@ public CellPath path()
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
- return new ArrayCell(newColumn, timestamp, ttl, localDeletionTime, value, path);
+ return new ArrayCell(newColumn, timestamp, ttl, Cell.deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
}
public Cell<?> withUpdatedValue(ByteBuffer newValue)
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, ByteBufferUtil.getArray(newValue), path);
+ return new ArrayCell(column, timestamp, ttl, Cell.deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), ByteBufferUtil.getArray(newValue), path);
}
- public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, int newLocalDeletionTime)
+ public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, long newLocalDeletionTime)
{
return new ArrayCell(column, newTimestamp, ttl, newLocalDeletionTime, value, path);
}
@Override
public Cell<?> withSkippedValue()
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, EMPTY_BYTE_ARRAY, path);
+ return new ArrayCell(column, timestamp, ttl, Cell.deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), EMPTY_BYTE_ARRAY, path);
Review Comment:
```suggestion
return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), EMPTY_BYTE_ARRAY, path);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107030552
##########
test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java:
##########
@@ -313,16 +314,18 @@ public void testCompactionInvalidTombstone() throws Throwable
{
DatabaseDescriptor.setCorruptedTombstoneStrategy(Config.CorruptedTombstoneStrategy.exception);
prepare();
+
// write a standard tombstone with negative local deletion time (LDTs are not set by user and should not be negative):
- RowUpdateBuilder rub = new RowUpdateBuilder(getCurrentColumnFamilyStore().metadata(), -1, System.currentTimeMillis() * 1000, 22).clustering(33).delete("b");
- rub.build().apply();
- flush();
- compactAndValidate();
- readAndValidate(true);
- readAndValidate(false);
+ Assertions.assertThatThrownBy(() -> {
+ new RowUpdateBuilder(getCurrentColumnFamilyStore().metadata(),
+ -1,
+ System.currentTimeMillis() * 1000,
+ 22).clustering(33).delete("b");
+ }).isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("is outside the range for an unsigned integer value");
}
- @Test
+ @Test(expected = RuntimeException.class) // msg = "is outside the range for an unsigned integer value"
Review Comment:
This means that now it fails on flush, so instead of generating a corrupt sstable we are not generating the sstable at all, right? Would all other writes for the same sstable be rejected too? Can this happen on a real use case?
By the way, the test could end in the call to `flush`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106851140
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -108,7 +127,7 @@ public ByteBuffer buffer()
* @return the cell local deletion time, or {@code NO_DELETION_TIME} if the cell is neither
* a tombstone nor an expiring one.
*/
- public abstract int localDeletionTime();
+ public abstract long localDeletionTime();
Review Comment:
So the implementations of `Cell` will now _still_ have to implement one method? I don't see the gain unless I am missing sthg.
Adding a `NativeCell` int friendly constructor sound like a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106996520
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -598,10 +598,12 @@ private static void loadLegacyTable(String tablePattern, String legacyVersion) t
* <p>
* Run this test alone (e.g. from your IDE) when a new version is introduced or format changed
* during development. I.e. remove the {@code @Ignore} annotation temporarily.
+ *
+ * *************RUN THIS TEST ALONE to avoid unwanted compactions, flushes, etc to interfere
* </p>
*/
@Ignore
- @Test
+ //@Test
Review Comment:
Why this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1532595301
@adelapena I just pushed some commits with a -D feature flag where:
- You can start 5.0 nodes with the nc and 2038y limit
- Those nodes work in a mixed cluster with 4.1nc nodes
- Once you have upgraded all to 5.0nc you can upgrade them then to 5.0oa
- When the last node updates to 5.0oa the new 2106 limit kicks in.
- There are new junit-oa variants in circle and a new jvm upgrade dtest where everything [passes](https://app.circleci.com/pipelines/github/bereng/cassandra/926/workflows/a4bdf7c7-e2c7-43e3-b964-21efad32dfc2)
wdyt? Thx in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1184558842
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -69,11 +73,19 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
{
private final static Logger logger = LoggerFactory.getLogger(BigFormat.class);
+ // Feature flag env variable to use extended TTL up to 2106 with the new oa sstable format. See c14227
+ public static enum TTL_MODE {LEGACY_TTL_2038, COMPATIBILITY, EXTENDED_TTL_2106};
+ @VisibleForTesting
+ public static final String USE_OA_SSTABLE_FORMAT = "cassandra.use_oa_sstable_format";
+ private static final String ttlFeatureFlagValue = System.getProperty(USE_OA_SSTABLE_FORMAT, "").toUpperCase();
+ public final static TTL_MODE ttlMode = EnumUtils.getEnumMap(TTL_MODE.class).keySet().contains(ttlFeatureFlagValue) ? TTL_MODE.valueOf(ttlFeatureFlagValue)
Review Comment:
Yeah, I was trying not to throw on an invalid value and go for the default. But I think it makes more sense to just fail upon an invalid Enum value here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185888261
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,16 +70,37 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToUnsignedInteger(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? NO_DELETION_TIME_UNSIGNED_INTEGER : CassandraUInt.fromLong(deletionTime);
+ }
+
+ public static long deletionTimeUnsignedIntegerToLong(int deletionTimeUnsignedInteger)
+ {
+ return deletionTimeUnsignedInteger == NO_DELETION_TIME_UNSIGNED_INTEGER ? NO_DELETION_TIME : CassandraUInt.toLong(deletionTimeUnsignedInteger);
+ }
+
+ public static long getVersionedMaxDeletiontionTime()
+ {
+ if (BigFormat.ttlMode.equals(TTL_MODE.EXTENDED_TTL_2106))
Review Comment:
```suggestion
if (BigFormat.ttlMode == TTL_MODE.EXTENDED_TTL_2106)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1186386170
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,16 +70,37 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToUnsignedInteger(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? NO_DELETION_TIME_UNSIGNED_INTEGER : CassandraUInt.fromLong(deletionTime);
+ }
+
+ public static long deletionTimeUnsignedIntegerToLong(int deletionTimeUnsignedInteger)
+ {
+ return deletionTimeUnsignedInteger == NO_DELETION_TIME_UNSIGNED_INTEGER ? NO_DELETION_TIME : CassandraUInt.toLong(deletionTimeUnsignedInteger);
+ }
+
+ public static long getVersionedMaxDeletiontionTime()
+ {
+ if (BigFormat.ttlMode.equals(TTL_MODE.EXTENDED_TTL_2106))
+ // The whole cluster is 2016, we're out of the 2038/2106 mixed clusters scenario. Short cut to avoid the 'minClusterVersion' volatile read
Review Comment:
Agree on both things. Trying to infer the behaviour should only be done during upgrades, and the avoidance on the version read will miss the `MAX_TTL` case.
However, I still think that the property can easily be more generic, so we don't merge code that will need to be undone by [CASSANDRA-18301](https://issues.apache.org/jira/browse/CASSANDRA-18301). The compatibility mode property can be generalized to cover any similar features with minimal changes. It can also be put on its own class and made accessible through `DatabaseDescriptor`, so we don't overload `BigFormat` nor we have messaging, commitlog, hints, etc. depending on that particular sstable format.
I gave it a go here: https://github.com/adelapena/cassandra/commit/bfbbb449b2645cef80d7ed040593ff07492610ec
It's basically the same approach but with some minor refactoring and a new rolling upgrade test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1547390705
@adelapena I just gave it another go at moving the FF to yaml. The problem is when executing junit-oa and similar; junit statics correctly default to nc as many don't read the yaml or extend CQLTester. Hence I had to preserve an env variable just so that in CI we set up statics correctly. Let me know what you think. Thx in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1196775956
##########
conf/cassandra.yaml:
##########
@@ -1921,3 +1921,30 @@ sstable_formats:
parameters:
id: 0
name: big
+
+
+# This property indicates with what Cassandra major version the storage format will be compatible with.
+#
+# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
+#
+# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
+# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,
+# so that feature won't be available if this property is set to CASSANDRA_4. See upgrade guides for details.
+#
+# Possible values are in the StorageCompatibilityMode.java file accessible online. At the time of writing these are:
+# - CASSANDRA_Y: Stays compatible with the Y.x line in features, formats and component versions
Review Comment:
Nit: I would explicitly mention every supported major, which at the moment is only `CASSANDRA_4`. That way users wouldn't try to set this to a not-existing `CASSANDRA_3` mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1205042939
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
No I didn't mean to remove the auto detection at all. Just to have a single ant_repeat job but that doesn't allow to run multiple repeats in a single push as you say...
One problem I hit, iiuc, is that when specifying a test to repeat the auto detection would still kick in and add lots of tests I didn't want to repeat at the time. I thought it would be useful to add a flag to skip auto detection so I have a way to repeat only what I want. I added -s [wdyt](https://github.com/bereng/cassandra/commit/03842aa1b8dfd6149f08dbfdb52c3eeabe5d9330)?
As per the job renames and repeats here is an [example](https://app.circleci.com/pipelines/github/bereng/cassandra/946/workflows/baf437ce-7198-4c4c-9530-9d59b42760f8) on my drafts branch. I think all is ready now and we only need to wait for sstable tries to merge and rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1203530498
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
I was going to look into this and then decided everything is too nosy and cluttered already. We should use `REPEATED_ANT_TEST_TARGET` for all these variations imo.
At the beginning that was nice sugar-coating but now it's growing and it will only get bigger imo. Circle config files are painful enough. I would leave it as it is and open a ticket to drop all ant based repeats, use the single ant repeat always and document it properly :shrug: That should pay itself over time with the shortened config files. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1171477928
##########
src/java/org/apache/cassandra/io/sstable/format/big/RowIndexEntry.java:
##########
@@ -59,7 +59,7 @@
* (long) position (64 bit long, vint encoded)
* (int) serialized size of data that follows (32 bit int, vint encoded)
* -- following for indexed entries only (so serialized size > 0)
- * (int) DeletionTime.localDeletionTime
+ * (int) int/Uint DeletionTime.localDeletionTime
Review Comment:
```suggestion
* (int) int/Uint DeletionTime.localDeletionTime (uint if version >= "oa" (5.0))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210238363
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
+ * But many tests don't load the config hence we need to force it otherwise they would run always under the default.
+ *
+ * This is a dev/CI only property. Do not use otherwise.
Review Comment:
I meant exactly that one, that the default compatibility mode is null for some junits that don't load the config, so the compatibility mode is got from the env var that CI/build.xml sets. What other thing did you refer to with the "Done" comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1145738217
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -94,7 +94,7 @@
* See {@link #testGenerateSstables()} to generate sstables.
* Take care on commit as you need to add the sstable files using {@code git add -f}
*/
- public static final String[] legacyVersions = {"nb", "na", "me", "md", "mc", "mb", "ma"};
+ public static final String[] legacyVersions = {"oa", "nb", "na", "me", "md", "mc", "mb", "ma"};
Review Comment:
Started a conversation [here](https://lists.apache.org/thread/1skggvfqzsznrrq0o5ztc58oxowh1x0g) to gather details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129240955
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -242,47 +249,52 @@ void mergeNearestPoints()
{
assert isFull() : "DataHolder must be full in order to merge two points";
- final int[] smallestDifference = findPointPairWithSmallestDistance();
+ final long[] smallestDifference = findPointPairWithSmallestDistance();
- final int point1 = smallestDifference[0];
- final int point2 = smallestDifference[1];
+ final long point1 = smallestDifference[0];
+ final long point2 = smallestDifference[1];
- long key = wrap(point1, 0);
- int index = Arrays.binarySearch(data, key);
+ int index = Arrays.binarySearch(points, point1);
if (index < 0)
{
index = -index - 1;
- assert (index < data.length) : "Not found in array";
- assert (unwrapPoint(data[index]) == point1) : "Not found in array";
+ assert (index < points.length) : "Not found in array";
+ assert (points[index] == point1) : "Not found in array";
}
- long value1 = unwrapValue(data[index]);
- long value2 = unwrapValue(data[index + 1]);
+ long value1 = values[index];
+ long value2 = values[index + 1];
- assert (unwrapPoint(data[index + 1]) == point2) : "point2 should follow point1";
+ assert (points[index + 1] == point2) : "point2 should follow point1";
long sum = value1 + value2;
//let's evaluate in long values to handle overflow in multiplication
- int newPoint = saturatingCastToInt((point1 * value1 + point2 * value2) / sum);
+ //long newPoint = saturatingCastToMaxDeletionTime((point1 * value1 + point2 * value2) / sum); // TODO 14227 Better weighted merged point possible?
+ long newPoint = (point2 - point1) >> 1; // approx division by 2 for perf reasons
Review Comment:
It's not equivalent. That is the mid point that is later used for a simple average. At the time of writing that was the best that didn't break stuff. But you triggered me to revisit this again and with fresh eyes I pushed a weighted average version that works. I think when I originally tried that long ago I wasn't accounting for merge boundaries correctly if my memory serves me right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129262330
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
> I think the comment you're asking for is there [already](https://github.com/apache/cassandra/pull/1891/files#diff-dbf9d67e1f6060176920b0d2555d111ef2d0fc0c3c3a0bec8e49c907155194eeR331)?
That link sends me to the global diff of all files, so I'm not sure what comment you are referring to. If it's [this one|https://github.com/bereng/cassandra/blob/CASSANDRA-14227-trunk/test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java#L331-L332], the part that I'd change is explicitly mentioning that the commented code generates the sstable. As in `To generate the sstables with corrupted data, run this commented code in some pre-c14227 branch that allows for negative ldts`.
> The 'extra' rows you are seeing are being inserted by the '[prepare](https://github.com/apache/cassandra/pull/1891/files#diff-dbf9d67e1f6060176920b0d2555d111ef2d0fc0c3c3a0bec8e49c907155194eeR394)()' methods iiuc.
Oh, right, I missed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129268385
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -88,15 +91,15 @@ public StreamingTombstoneHistogramBuilder(int maxBinSize, int maxSpoolSize, int
*
* @param point the point to be added
*/
- public void update(int point)
+ public void update(long point)
Review Comment:
Great, thanks. There is another outdated reference to `SSTableReader#getDroppableTombstonesBefore(int)` on the class JavaDoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1132006460
##########
test/unit/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilderTest.java:
##########
@@ -138,7 +137,7 @@ public void testNumericTypes() throws Exception
asMap = asMap(deserialized);
assertEquals(1, deserialized.size());
- assertEquals(Integer.MAX_VALUE, asMap.get(2).intValue());
+ assertEquals(Integer.MAX_VALUE, asMap.get(2l).intValue());
Review Comment:
Will do
##########
test/unit/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilderTest.java:
##########
@@ -168,8 +167,8 @@ public void testRounding() throws Exception
builder.update(samples[i]);
TombstoneHistogram hist = builder.build();
assertEquals(hist.size(), 5);
- assertEquals(asMap(hist).get(60).intValue(), 2);
- assertEquals(asMap(hist).get(120).intValue(), 1);
+ assertEquals(asMap(hist).get(60l).intValue(), 2);
+ assertEquals(asMap(hist).get(120l).intValue(), 1);
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134438171
##########
src/java/org/apache/cassandra/db/compaction/Scrubber.java:
##########
@@ -614,7 +619,84 @@ public Unfiltered next()
}
nextToOffer = null;
- return next;
+ return computeFinalRow((Row) next);
+ }
+
+ private Row computeFinalRow(Row next)
+ {
+ // If the row has overflowed let rows skip them unless we need to keep them for the overflow policy
+ if (hasOverflowedLocalExpirationTimeRow((Row) next) && !reinsertOverflowedTTLRows)
+ return null;
+ else if (reinsertOverflowedTTLRows)
+ return rebuildTimestamptsForOverflowedRows(next);
+ else
+ return next;
+ }
+
+ /*
+ * When building ldt on deser it won't overflow now being a long as it used to. This causes row resurrection for old sstable formats!
+ * To prevent it we preserve the overflow to be backwards compatible and to feed into the overflow policy
+ */
+ private Row rebuildTimestamptsForOverflowedRows(Row row)
+ {
+ if (sstableVersion.hasLongLocalDeletionTime())
+ return row;
+
+ LivenessInfo livenessInfo = row.primaryKeyLivenessInfo();
+ if (livenessInfo.isExpiring() && livenessInfo.localExpirationTime() >= 0)
+ {
+ livenessInfo = livenessInfo.withUpdatedTimestampAndLocalDeletionTime(livenessInfo.timestamp(), livenessInfo.localExpirationTime(), false);
+ }
+
+ return row.transformAndFilter(livenessInfo, row.deletion(), cd -> {
+ if (cd.column().isSimple())
+ {
+ Cell cell = (Cell)cd;
Review Comment:
```suggestion
Cell<?> cell = (Cell<?>) cd;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by GitBox <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1269609749
Rebased + fixed the unrelated rolling upgrade test. Green [CI](https://app.circleci.com/pipelines/github/bereng/cassandra/793/workflows/c4836865-d106-4c7c-8f3f-d7b2df899aa0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210262943
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
+ * But many tests don't load the config hence we need to force it otherwise they would run always under the default.
+ *
+ * This is a dev/CI only property. Do not use otherwise.
Review Comment:
Don't you see that comment in the preceding paragraph?
```
...Config is null for junits that don't load the config. Get from env var that CI/build.xml sets.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1120191633
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
This adds a `ks` parameter instead of using `"legacy_tables"`. However, the passed `ks` argument is always `"legacy_tables"`, so the change seems unneeded.
Perhaps we could replace the 21 repetitions of the string `"legacy_tables"` by a `static final` class constant, make this method `private` and remove the parameter:
```java
private static final String LEGACY_TABLES_KEYSPACE = "legacy_tables";
...
private static void copySstablesFromTestData(String table, File ksDir) throws IOException
{
File cfDir = new File(ksDir, table);
cfDir.tryCreateDirectory();
for (File srcDir : Keyspace.open(LEGACY_TABLES_KEYSPACE).getColumnFamilyStore(table).getDirectories().getCFDirectories())
{
for (File file : srcDir.tryList())
{
copyFile(cfDir, file);
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1119624336
##########
test/unit/org/apache/cassandra/cql3/validation/entities/SecondaryIndexTest.java:
##########
@@ -1575,7 +1575,7 @@ private void testIndexOnPartitionKeyInsertExpiringColumn(boolean flushBeforeUpda
flush();
execute("UPDATE %s USING TTL 1 SET b = 10 WHERE k1 = 1 AND k2 = 2");
- Thread.sleep(1000);
+ Thread.sleep(1500);
Review Comment:
Reverted in latest commit in case it was a side effect of one of my changes at the time during dev. I'll open a ticket if it surfaces again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1449755141
A good policy is that reviewers’ time is more valuable than your own - and expecting them to expand every resolved comment to check it was correctly closed, particularly when there are multiple rounds of feedback, is burdensome.
I haven’t provided much feedback here, so I’m not bothered, I’m just alerting you to the fact you should consider this going forwards. I would care if I were performing a proper review and I know others would too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1449746183
It's also extra burden and distracting to have a neverending lists of unresolved comments, at least for me. 99% of the comments I resolved haven't come back so it's a good hit rate so far. There's no silver bullet I'm afraid. I did ping Andres to let me know if it was a problem but I didn't ask you. I'll wait for you to resolve then, let me know if I you want me to unresolve anything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107151986
##########
src/java/org/apache/cassandra/db/LivenessInfo.java:
##########
@@ -60,18 +60,33 @@ protected LivenessInfo(long timestamp)
this.timestamp = timestamp;
}
- public static LivenessInfo create(long timestamp, int nowInSec)
+ public static LivenessInfo create(long timestamp, long nowInSec)
{
return new LivenessInfo(timestamp);
}
- public static LivenessInfo expiring(long timestamp, int ttl, int nowInSec)
+ public static LivenessInfo expiring(long timestamp, int ttl, long nowInSec)
{
assert ttl != EXPIRED_LIVENESS_TTL;
return new ExpiringLivenessInfo(timestamp, ttl, ExpirationDateOverflowHandling.computeLocalExpirationTime(nowInSec, ttl));
}
- public static LivenessInfo create(long timestamp, int ttl, int nowInSec)
+ // c14227 do not use. Special use to circumvent the overflow policy when scrubbing
+ public static LivenessInfo expiring(long timestamp, int ttl, long nowInSec, boolean applyOverflowPolicy)
Review Comment:
Can be `private`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106804057
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
And I totally agree bitwise operations are not dangerous or exotic. But I'd rather run hot needles through my eyes than removing that and have to deal with such bugs in the future I've had to deal with when it wasn't present.
It does provide protection which has proven invaluable.
```
java.lang.IllegalArgumentException: value (-207332) is outside the range for an unsigned integer value
[junit-timeout] at com.google.common.base.Preconditions.checkArgument(Preconditions.java:202)
[junit-timeout] at com.google.common.primitives.UnsignedInteger.valueOf(UnsignedInteger.java:75)
[junit-timeout] at org.apache.cassandra.db.rows.Cell.deletionTimeLongToGuavaUint(Cell.java:81)
```
It's an accident waiting to happen, we're shooting ourselves in the foot imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107042468
##########
test/unit/org/apache/cassandra/db/NativeCellTest.java:
##########
@@ -125,7 +125,7 @@ private static Cell<?> rndcell(ColumnMetadata col)
{
long timestamp = rand.nextLong();
int ttl = rand.nextInt();
- int localDeletionTime = rand.nextInt();
+ long localDeletionTime = Math.abs(rand.nextInt());
Review Comment:
Maybe this would be clearer with `Cell.deletionTimeGuavaUintToLong(rand.nextInt())`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108219623
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,10 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
+ // Guava uint counterpart to the above for serialization purposes. This imposes a year 2186 max value
+ private static final int UNSET_NOWINSEC_GUAVA_UINT = UnsignedInteger.MAX_VALUE.intValue() - 1;
Review Comment:
You want to protect against `UnsignedInteger.MAX_VALUE` becoming/being a sentinel value, so -1 should do the trick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107075848
##########
src/java/org/apache/cassandra/net/OutboundConnection.java:
##########
@@ -1127,7 +1127,7 @@ void onCompletedHandshake(Result<MessagingSuccess> result)
FrameEncoder.PayloadAllocator payloadAllocator = success.allocator;
Channel channel = success.channel;
- Established established = new Established(messagingVersion, channel, payloadAllocator, settings);
+ Established established = new Established(success.messagingVersion, channel, payloadAllocator, settings);
Review Comment:
Why this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1104865054
##########
src/java/org/apache/cassandra/db/rows/BufferCell.java:
##########
@@ -34,21 +34,22 @@ public class BufferCell extends AbstractCell<ByteBuffer>
{
private static final long EMPTY_SIZE = ObjectSizes.measure(new BufferCell(ColumnMetadata.regularColumn("", "", "", ByteType.instance), 0L, 0, 0, ByteBufferUtil.EMPTY_BYTE_BUFFER, null));
+ // Careful: Adding vars here has an impact on memtable size
private final long timestamp;
private final int ttl;
- private final int localDeletionTime;
+ private final int localDeletionTimeGuavaUint;
private final ByteBuffer value;
private final CellPath path;
- public BufferCell(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, ByteBuffer value, CellPath path)
+ public BufferCell(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, ByteBuffer value, CellPath path)
{
super(column);
assert !column.isPrimaryKeyColumn();
assert column.isComplex() == (path != null) : format("Column %s.%s(%s: %s) isComplex: %b with cellpath: %s", column.ksName, column.cfName, column.name, column.type.toString(), column.isComplex(), path);
this.timestamp = timestamp;
this.ttl = ttl;
- this.localDeletionTime = localDeletionTime;
+ this.localDeletionTimeGuavaUint = Cell.deletionTimeLongToGuavaUint(localDeletionTime);
Review Comment:
```suggestion
this.localDeletionTimeGuavaUint = Cell.deletionTimeLongToGuavaUint(localDeletionTime);
```
```suggestion
this.localDeletionTimeGuavaUint = deletionTimeLongToGuavaUint(localDeletionTime);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105731270
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
It really shouldn't. The cast is free, you just load into a suitably sized register. The truncation will have a single cycle latency (and greater than one-per-cycle throughput). If you can measure the impact of this in a real C* trace, I will eat my hat. If you can even get it to show up in a microbenchmark that does _anything_ real (compaction, memtable read or anything really) I will be very impressed. It's going to be 100% swallowed up by memory latency, data dependencies in the pipeline, etc.
However, the wasted memory will definitely have an impact. For many JVMs this will require an extra 8-bytes of memory.
So this is a pretty simple tradeoff, and I would not personally waste my time benchmarking it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105701234
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
Converting uint<-> long should have some perf impact hence having it readily available as a long it's nice. But given I do indeed store an int for [Cells](https://github.com/apache/cassandra/pull/1891/files/449129ca26b83efed084471d822a52f81cb3a0c6#diff-bdff7d68605bca83e303d541e5c301d289c2b0fd54a9b04e2185ffc5d2ca8770R42) which is the same optimization i.e., I should perf test moving this to an int and see if it has an impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106058178
##########
src/java/org/apache/cassandra/db/rows/ArrayCell.java:
##########
@@ -83,23 +86,23 @@ public CellPath path()
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
- return new ArrayCell(newColumn, timestamp, ttl, localDeletionTime, value, path);
+ return new ArrayCell(newColumn, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
}
public Cell<?> withUpdatedValue(ByteBuffer newValue)
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, ByteBufferUtil.getArray(newValue), path);
+ return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), ByteBufferUtil.getArray(newValue), path);
}
- public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, int newLocalDeletionTime)
+ public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, long newLocalDeletionTime)
Review Comment:
This can just call to `localDeletionTime()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1111595605
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +45,14 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ // -5 to leave room for any future sentinel values just in case
Review Comment:
You got me thinking that if we don't like this approach, when/if the time comes we need it we could use times in the past before the C* days as sentinel values. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1110052763
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +45,14 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ // -5 to leave room for any future sentinel values just in case
Review Comment:
Do we have any idea about what those future values would be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1110057431
##########
test/unit/org/apache/cassandra/db/NativeCellTest.java:
##########
@@ -125,7 +125,7 @@ private static Cell<?> rndcell(ColumnMetadata col)
{
long timestamp = rand.nextLong();
int ttl = rand.nextInt();
- int localDeletionTime = rand.nextInt();
+ long localDeletionTime = Math.abs(rand.nextInt());
Review Comment:
It could be useful to leave this unresolved until the change is pushed, so we don't forget it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122661198
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +45,14 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ // -5 to leave room for any future sentinel values just in case
Review Comment:
MAX - 1 is `LEGACY_OVERFLOWED_DELETION_TIME` right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122642371
##########
test/unit/org/apache/cassandra/utils/CassandraGenerators.java:
##########
@@ -99,7 +100,7 @@
private static final Gen<SinglePartitionReadCommand> SINGLE_PARTITION_READ_COMMAND_GEN = gen(rnd -> {
TableMetadata metadata = TABLE_METADATA_GEN.generate(rnd);
- int nowInSec = (int) rnd.next(Constraint.between(1, Integer.MAX_VALUE));
+ long nowInSec = rnd.next(Constraint.between(1, Cell.MAX_DELETION_TIME));
Review Comment:
Bc it's the max that makes sense? What would you suggest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1173382984
##########
NEWS.txt:
##########
@@ -1,9 +1,10 @@
-PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092)
-------------------------------------------------------------------
+PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092 & CASSANDRA-14227)
+-----------------------------------------------------------------------------------
(General upgrading instructions are available in the next section)
-The maximum expiration timestamp that can be represented by the storage engine is
-2038-01-19T03:14:06+00:00, which means that inserts with TTL thatl expire after
+The maximum expiration timestamp that can be represented by the storage engine has been
+raised to 2106-02-07T06:28:13+00:00 (2038-01-19T03:14:06+00:00 if in compatibility mode),
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127548434
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
I don't have a better idea here. I'll revert. Can you please resolve all the rest you're happy with?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129286429
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -242,47 +249,51 @@ void mergeNearestPoints()
{
assert isFull() : "DataHolder must be full in order to merge two points";
- final int[] smallestDifference = findPointPairWithSmallestDistance();
+ final long[] smallestDifference = findPointPairWithSmallestDistance();
- final int point1 = smallestDifference[0];
- final int point2 = smallestDifference[1];
+ final long point1 = smallestDifference[0];
+ final long point2 = smallestDifference[1];
- long key = wrap(point1, 0);
- int index = Arrays.binarySearch(data, key);
+ int index = Arrays.binarySearch(points, point1);
if (index < 0)
{
index = -index - 1;
- assert (index < data.length) : "Not found in array";
- assert (unwrapPoint(data[index]) == point1) : "Not found in array";
+ assert (index < points.length) : "Not found in array";
+ assert (points[index] == point1) : "Not found in array";
}
- long value1 = unwrapValue(data[index]);
- long value2 = unwrapValue(data[index + 1]);
-
- assert (unwrapPoint(data[index + 1]) == point2) : "point2 should follow point1";
+ long value1 = values[index];
+ long value2 = values[index + 1];
- long sum = value1 + value2;
+ assert (points[index + 1] == point2) : "point2 should follow point1";
- //let's evaluate in long values to handle overflow in multiplication
- int newPoint = saturatingCastToInt((point1 * value1 + point2 * value2) / sum);
+ long a = saturatingCastToLong(point1 * value1);
+ long b = saturatingCastToLong(point2 * value2);
+ long sum = saturatingCastToLong(value1 + value2);
+ long newPoint = saturatingCastToMaxDeletionTime(saturatingCastToLong(a + b) / sum);
+ newPoint = newPoint <= point1 ? saturatingCastToMaxDeletionTime(point1 + 1) : newPoint;
+ newPoint = newPoint > point2 ? point2 : newPoint;
Review Comment:
```suggestion
newPoint = Math.min(newPoint, point2);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1132003427
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -88,15 +91,15 @@ public StreamingTombstoneHistogramBuilder(int maxBinSize, int maxSpoolSize, int
*
* @param point the point to be added
*/
- public void update(int point)
+ public void update(long point)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135039803
##########
src/java/org/apache/cassandra/db/compaction/Scrubber.java:
##########
@@ -614,7 +619,84 @@ public Unfiltered next()
}
nextToOffer = null;
- return next;
+ return computeFinalRow((Row) next);
+ }
+
+ private Row computeFinalRow(Row next)
+ {
+ // If the row has overflowed let rows skip them unless we need to keep them for the overflow policy
+ if (hasOverflowedLocalExpirationTimeRow((Row) next) && !reinsertOverflowedTTLRows)
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134057932
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,17 +42,41 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
Review Comment:
Can be `private`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1479151926
@adelapena I notice [here](https://github.com/apache/cassandra/commit/7ca806c60a3e080d740fb163c639bb76a520f6ab#diff-7284e8da2bb0ea3e0e965a8f728407b2326f982281d6b37ca0d7cad79039b70eL214), this is Mick's ticket moving trunk to be v5.0, he removes messaging version 41. So I think I have a bug in my PR where I added v41 and v42 tying the DB version bump to a messaging version bump. I should only add one more version correct? This area is a bit fuzzy for me apologies if I am missing sthg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1144679045
##########
CHANGES.txt:
##########
@@ -1,4 +1,6 @@
+<<<<<<< HEAD
Review Comment:
Nit: this seems a leftover of solving merge conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1187111232
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,16 +70,37 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToUnsignedInteger(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? NO_DELETION_TIME_UNSIGNED_INTEGER : CassandraUInt.fromLong(deletionTime);
+ }
+
+ public static long deletionTimeUnsignedIntegerToLong(int deletionTimeUnsignedInteger)
+ {
+ return deletionTimeUnsignedInteger == NO_DELETION_TIME_UNSIGNED_INTEGER ? NO_DELETION_TIME : CassandraUInt.toLong(deletionTimeUnsignedInteger);
+ }
+
+ public static long getVersionedMaxDeletiontionTime()
+ {
+ if (BigFormat.ttlMode.equals(TTL_MODE.EXTENDED_TTL_2106))
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185720551
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -69,11 +73,19 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
{
private final static Logger logger = LoggerFactory.getLogger(BigFormat.class);
+ // Feature flag env variable to use extended TTL up to 2106 with the new oa sstable format. See c14227
+ public static enum TTL_MODE {LEGACY_TTL_2038, COMPATIBILITY, EXTENDED_TTL_2106};
+ @VisibleForTesting
+ public static final String USE_OA_SSTABLE_FORMAT = "cassandra.use_oa_sstable_format";
Review Comment:
That I like a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185713831
##########
src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java:
##########
@@ -62,13 +64,14 @@
// We don't support anything pre-3.0
public static final int VERSION_30 = 6;
public static final int VERSION_40 = 7;
+ public static final int VERSION_50 = 8;
/**
* Increment this number if there is a changes in the commit log disc layout or MessagingVersion changes.
* Note: make sure to handle {@link #getMessagingVersion()}
*/
@VisibleForTesting
- public static final int current_version = VERSION_40;
+ public static final int current_version = BigFormat.ttlMode.equals(TTL_MODE.LEGACY_TTL_2038) ? VERSION_40 : VERSION_50;
Review Comment:
Ops I missed those 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1196777173
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
Review Comment:
Nit: the link tag seem malformed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1203857352
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
Ok I can rename, no problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1209708978
##########
conf/cassandra.yaml:
##########
@@ -1921,3 +1921,30 @@ sstable_formats:
parameters:
id: 0
name: big
+
+
+# This property indicates with what Cassandra major version the storage format will be compatible with.
+#
+# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
+#
+# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
+# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,
+# so that feature won't be available if this property is set to CASSANDRA_4. See upgrade guides for details.
+#
+# Possible values are in the StorageCompatibilityMode.java file accessible online. At the time of writing these are:
+# - CASSANDRA_Y: Stays compatible with the Y.x line in features, formats and component versions
Review Comment:
Done
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1567790450
@adelapena this is rebased as of yesterday:
J8/J11
- Perf testing can be found [here](https://issues.apache.org/jira/secure/attachment/13058571/13058571_C14227+Perf+check+2023.05.26.pdf)
- I went over all touched tests manually and 90% are just simple int to long changes. The only tests worth repeating are MixedModeTTLOverflowUpgradeTest, CustomNowInSecondsTest, EmptyValuesTest, TTLTest, CellTest, RangeTombstoneListTest, CompactionsCQLTest, LegacySSTableTest, PaxosStateTest and StreamingTombstoneHistogramBuilderTest.
- [j8](https://app.circleci.com/pipelines/github/bereng/cassandra/957/workflows/6a9138af-7950-4f78-937a-51b173197477) seems ok: test_rolling_upgrade_with_internode_ssl is CASSANDRA-18449. test_fql_nodetool_options is a nit mistake of mine with dtests that I fixed before running the j11 which you'll see is all green
- j8 includes 500 repetitions in junit, junit_oa and junit_system_keyspace. That is 1500 repetitions
- [j11](https://app.circleci.com/pipelines/github/bereng/cassandra/957/workflows/a1fa869d-7426-431f-8c49-59ad45fb9431) is green
- 500 repetitions of the new upgrade jvm tests [is green](https://app.circleci.com/pipelines/github/bereng/cassandra/958/workflows/784d60b8-71b7-4447-8be4-58c8b7f4e655)
J11/J17
- J11 [here](https://app.circleci.com/pipelines/github/bereng/cassandra/959/workflows/9598d280-1121-4a6f-ad4a-3cd96ee20b44). Failures aligned to a baseline trunk [j17 run](https://app.circleci.com/pipelines/github/bereng/cassandra/961/workflows/3e073e0f-c43c-4dea-be51-44e911b41aa9). handleCorruptionOfLargeMessageFrame is CASSANDRA-16949
- We can see the oa job variants in j11 and j17
- J17 [run](https://app.circleci.com/pipelines/github/bereng/cassandra/959/workflows/fdf2aae9-6970-42e1-a8d6-785adc656229) is aligned to J17 baseline
Unless I am missing sthg this is ready to be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210165953
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
Review Comment:
It has been changed to `{@link StorageCompatibility}`, it should be `{@link StorageCompatibilityMode}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1576040158
Well I thought about that detail. But you know I see all that as part of C18301. In fact all this feature flag I would have left for c18301. So I still think that's the right thing to do. Having said that I won't oppose opening a small ticket to address it if that's your preference. Now lets get this merged, correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1188502055
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
You have my +1 on the approach :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135337982
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +255,31 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ private InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
Review Comment:
Great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135037160
##########
src/java/org/apache/cassandra/utils/CassandraUInt.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.utils;
+
+import com.google.common.primitives.UnsignedInteger;
+import com.google.common.primitives.UnsignedInts;
+
+/**
+ * This class packages several UInt utility methods so we can easily change it's implementation when needed.
+ */
+public class CassandraUInt
+{
+ /**
+ * Returns the long representation of the maximum value we can hold
+ */
+ public final static long MAX_VALUE_LONG = UnsignedInteger.MAX_VALUE.longValue();
+ /**
+ * Returns the UInt representation of the maximum value we can hold
+ */
+ public final static int MAX_VALUE_UINT = UnsignedInteger.MAX_VALUE.intValue();
+
+ /**
+ * Converts a long to it's unsigned integer representation
+ *
+ * @param value A long between 0 and 2<sup>32</sup>-1 inclusive
+ * @return an unsigned integer representation of the long
+ * @throws IllegalArgumentException if value '< 0' or '>= 2<sup>32</sup>'
+ */
+ public static int fromLong(long value)
+ {
+ return UnsignedInts.checkedCast(value);
+ }
+
+ /**
+ * The same but works on an array on longs
+ *
+ */
+ public static int[] fromLong(long[] values)
+ {
+ int[] delTimesUints = new int[values.length];
+ for (int i=0; i < values.length; i++)
+ delTimesUints[i] = CassandraUInt.fromLong(values[i]);
+ return delTimesUints;
+ }
+
+ /**
+ * Returns the long resulting from parsing the int as an unsigned integer
+ *
+ * @param value Unsigned integer representation of the long
+ * @return The long resulting from parsing the int as an unsigned integer
+ */
+ public static long toLong(int value)
+ {
+ return UnsignedInts.toLong(value);
+ }
+
+ /**
+ * Compare 2 Uints
+ *
+ * @return < 0 if x is less than y. > 0 if x is greater than y. Zero if they are equal
+ */
+ public static int compare (int x, int y)
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135035332
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,26 +60,31 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private int[] delTimesUnsignedIntegers;
private long boundaryHeapSize;
private int size;
- private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, int[] delTimes, long boundaryHeapSize, int size)
+ private RangeTombstoneList(ClusteringComparator comparator, ClusteringBound<?>[] starts, ClusteringBound<?>[] ends, long[] markedAts, long[] delTimes, long boundaryHeapSize, int size)
Review Comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1187201435
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,16 +70,37 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToUnsignedInteger(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? NO_DELETION_TIME_UNSIGNED_INTEGER : CassandraUInt.fromLong(deletionTime);
+ }
+
+ public static long deletionTimeUnsignedIntegerToLong(int deletionTimeUnsignedInteger)
+ {
+ return deletionTimeUnsignedInteger == NO_DELETION_TIME_UNSIGNED_INTEGER ? NO_DELETION_TIME : CassandraUInt.toLong(deletionTimeUnsignedInteger);
+ }
+
+ public static long getVersionedMaxDeletiontionTime()
+ {
+ if (BigFormat.ttlMode.equals(TTL_MODE.EXTENDED_TTL_2106))
+ // The whole cluster is 2016, we're out of the 2038/2106 mixed clusters scenario. Short cut to avoid the 'minClusterVersion' volatile read
Review Comment:
Thx for that commit. I have shamelessly stolen it with some fixed test failures. Some comments:
- I like the 'generic' spin you gave it which will be nice for 18301
- I like it respects the tri-state. Here we're avoiding a volatile read. But if 18301 goes TCM or anything else, 99% probability it will be more expensive than a volatile read, we need a mechanism to avoid that cost in the interim upgrading state.
- I like it respects the idea of signaling the upgrade having finished and not having a window of opportunity to go back/forth accidentally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1184537595
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -69,11 +73,19 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
{
private final static Logger logger = LoggerFactory.getLogger(BigFormat.class);
+ // Feature flag env variable to use extended TTL up to 2106 with the new oa sstable format. See c14227
+ public static enum TTL_MODE {LEGACY_TTL_2038, COMPATIBILITY, EXTENDED_TTL_2106};
+ @VisibleForTesting
+ public static final String USE_OA_SSTABLE_FORMAT = "cassandra.use_oa_sstable_format";
+ private static final String ttlFeatureFlagValue = System.getProperty(USE_OA_SSTABLE_FORMAT, "").toUpperCase();
+ public final static TTL_MODE ttlMode = EnumUtils.getEnumMap(TTL_MODE.class).keySet().contains(ttlFeatureFlagValue) ? TTL_MODE.valueOf(ttlFeatureFlagValue)
+ : TTL_MODE.LEGACY_TTL_2038;
+
public static final BigFormat instance = new BigFormat();
private final Version latestVersion = new BigVersion(this, BigVersion.current_version);
private final BigTableReaderFactory readerFactory = new BigTableReaderFactory();
- private final BigTableWriterFactory writerFactory = new BigTableWriterFactory();
+ private final BigTableWriterFactory writerFactory = new BigTableWriterFactory();
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1196809343
##########
test/distributed/org/apache/cassandra/distributed/upgrade/MixedModeTTLOverflowUpgradeTest.java:
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.distributed.upgrade;
+
+import org.junit.Test;
+
+import org.apache.cassandra.cql3.Attributes;
+import org.apache.cassandra.db.rows.Cell;
+import org.apache.cassandra.distributed.api.ICoordinator;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.StorageCompatibilityMode;
+import org.assertj.core.api.Assertions;
+
+import static org.apache.cassandra.distributed.api.ConsistencyLevel.ALL;
+import static org.apache.cassandra.utils.StorageCompatibilityMode.NONE;
+import static org.apache.cassandra.utils.StorageCompatibilityMode.UPGRADING;
+
+/**
+ * Tests TTL the overflow policy triggers at the correct limit: year 2038 <=nc or 2186 >nc
+ * <p>
+ * <=nc overflow policy triggers at year 2038. That could be <=4.1 or 5.0 with 4.x storage compatibility
+ * >nc overflow policy triggers at year 2106. That is >=5.0 using >=5.x storage compatibility
+ *
+ * @see StorageCompatibilityMode
+ */
+public class MixedModeTTLOverflowUpgradeTest extends UpgradeTestBase
+{
+ @Test
+ public void testTTLOverflowDuringUpgrade() throws Throwable
+ {
+ testTTLOverflow((cluster, node) -> {
+ if (node == 1) // only node1 is upgraded, and the cluster is in mixed versions mode
+ {
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+
+ // We restart the upgraded node out of 4.0 storage compatibility,
+ // and we set it to be compatible with 5.0.
+ // 2038 should still be the limit because there is still a not upgraded node.
+ cluster.get(1).shutdown().get();
+ cluster.get(1).config().set("storage_compatibility_mode", UPGRADING.toString());
+ cluster.get(1).startup();
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+ }
+ else // both nodes have been upgraded, and the cluster isn't in mixed version mode anymore
+ {
+ // Once we have completed the upgrade, 2038 should still be the limit because there is still one node
+ // in 5.0 storage compatibility mode.
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+
+ // We restart the last upgraded node in 5.0 compatibility mode, so both nodes are now in 5.0
+ // compatibility mode, and the limit should be 2106.
+ cluster.get(2).shutdown().get();
+ cluster.get(2).config().set("storage_compatibility_mode", UPGRADING.toString());
+ cluster.get(2).startup();
+ assertPolicyTriggersAt2106(cluster.coordinator(1));
+ assertPolicyTriggersAt2106(cluster.coordinator(2));
+
+ // We restart get both nodes out of compatibility mode, so the limit should be 2106.
+ cluster.get(1).shutdown().get();
+ cluster.get(1).config().set("storage_compatibility_mode", NONE.toString());
+ cluster.get(1).startup();
+ cluster.get(2).shutdown().get();
+ cluster.get(2).config().set("storage_compatibility_mode", NONE.toString());
+ cluster.get(2).startup();
+ assertPolicyTriggersAt2106(cluster.coordinator(1));
+ assertPolicyTriggersAt2106(cluster.coordinator(2));
+ }
+ });
+ }
+
+ @Test
+ public void testTTLOverflowAfterUpgrade() throws Throwable
+ {
+ testTTLOverflow((cluster, node) -> {
+ if (node == 1) // only node1 is upgraded, and the cluster is in mixed versions mode
+ {
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+ }
+ else // both nodes have been upgraded, and the cluster isn't in mixed version mode anymore
+ {
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+
+ // We restart one node on 5.0 >nc hence 2038 should still be the limit as the other node is 5.0 <=nc
+ // We're on compatibility mode where nc and oa nodes are a possibility
+ cluster.get(1).shutdown().get();
+ cluster.get(1).config().set("storage_compatibility_mode", UPGRADING.toString());
+ cluster.get(1).startup();
+ assertPolicyTriggersAt2038(cluster.coordinator(1));
+ assertPolicyTriggersAt2038(cluster.coordinator(2));
+
+ // We restart the other node so they're all on 5.0 >nc hence 2106 should be the limit
+ cluster.get(2).shutdown().get();
+ cluster.get(2).config().set("storage_compatibility_mode", UPGRADING.toString());
+ cluster.get(2).startup();
+ assertPolicyTriggersAt2106(cluster.coordinator(1));
+ assertPolicyTriggersAt2106(cluster.coordinator(2));
+
+ // We restart the cluster out of compatibility mode once everything is 5.0oa TTL 2106
+ cluster.get(1).shutdown().get();
+ cluster.get(1).config().set("storage_compatibility_mode", NONE.toString());
+ cluster.get(1).startup();
+ cluster.get(2).shutdown().get();
+ cluster.get(2).config().set("storage_compatibility_mode", NONE.toString());
+ cluster.get(2).startup();
+ assertPolicyTriggersAt2106(cluster.coordinator(1));
+ assertPolicyTriggersAt2106(cluster.coordinator(2));
+ }
+ });
+ }
+
+ private static void testTTLOverflow(RunOnClusterAndNode runAfterNodeUpgrade) throws Throwable
+ {
+ new TestCase()
+ .nodes(2)
+ .nodesToUpgradeOrdered(1, 2)
+ .upgradesToCurrentFrom(v41)
Review Comment:
Strictly this should be at least `v40`, which also includes `v41`. There is a risk of OOM when doubling the number of tests, though. If that happens this can be split in two classes, inheriting from a common superclass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1517429666
After talking offline we decided the best way forward would be to provide here some feature flag [CASSANDRA-18301](https://issues.apache.org/jira/browse/CASSANDRA-18301) could piggyback upon or replace. Hence I'll add that with the underlying assumption mixed version clusters (5.0 nc <-> 5.0 oa) will always communicate on the lower common MessagingVersion in the cluster keeping things coherent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1173382131
##########
test/unit/org/apache/cassandra/cql3/CustomNowInSecondsTest.java:
##########
@@ -73,6 +79,48 @@ private void testSelectQuery(boolean prepared)
assertEquals(0, executeSelect(format("SELECT * FROM %s.%s", ks, tbl), now + day + 1, prepared).size());
}
+ @Test
+ public void testSelectQueryWitInthOverflow()
+ {
+ testSelectQueryOverflowingIntTimestamps(false);
+ testSelectQueryOverflowingIntTimestamps(true);
+ }
+
+ private void testSelectQueryOverflowingIntTimestamps(boolean prepared)
+ {
+ ExpirationDateOverflowPolicy origPolicy = ExpirationDateOverflowHandling.policy;
+ ExpirationDateOverflowHandling.policy = ExpirationDateOverflowPolicy.REJECT;
+
+ int ttl = Attributes.MAX_TTL - 10; // Give it a TTL that should overflow 'int' timestamps
+
+ String ks = createKeyspace("CREATE KEYSPACE %s WITH replication={ 'class' : 'SimpleStrategy', 'replication_factor' : 1 }");
+ String tbl = createTable(ks, "CREATE TABLE %s (id int primary key, val int)");
+
+ // insert a row with an int overflowing timestamp. Behavior will depend on versions
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1514364495
@adelapena I just committed:
- [fce297e](https://github.com/apache/cassandra/pull/1891/commits/fce297e17d5c7ad337912dad6bc6964c4b300b4f) to add the right 5.0 version
- [575f809](https://github.com/apache/cassandra/pull/1891/commits/575f809b57dde4c19ca682b0f06315a38d11f6c5) makes the code behave in 'compatibility mode': nc, MV 4.0, etc It [passes](https://app.circleci.com/pipelines/github/bereng/cassandra/913/workflows/8ec1fa9a-1c45-436a-8c22-2f20675fce5f) all tests. The only 2 dtest failures are bc on dtests we can't detect when we run in one mode or the other, sthg CASSANDRA-18301 should address. The tests would pass if that condition was detected. I also copied the a data folder into a 4.1 install and it works where the only errors were related to other tickets that again CASSANDRA-18301 should address
- [45aec2c](https://github.com/apache/cassandra/pull/1891/commits/45aec2cc5be195643b90ab61b95316ffdd19fda3) Switches the code back to a 5.0 config and all tests [pass](https://app.circleci.com/pipelines/github/bereng/cassandra/914/workflows/a2e9babe-8231-4ef9-9785-fc72ab887c31)
CASSANDRA-18301 should only toggle those versions and the codebase now behaves with the old or new behavior accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1203530498
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
I was going to look into this and then decided everything is too nosy and cluttered already. We should use `REPEATED_ANT_TEST_TARGET` for all these variations imo.
At the beginning that was nice sugar-coating but now it's growing and it will only get bigger imo. Circle config files are painful enough. I would leave it as it is and open a ticket to drop all specialized ant based repeats, use the generic ant repeat instead and document it properly :shrug: That should pay itself over time with the shortened config files, simpler diagrams, etc.
At the end of the day we want to put all the complexity into build.xml and exercise those through ant targets. We don't need to mirror that complexity in circle configs and have duplicate effort. :thinking:
Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1415254250
CI [j8](https://app.circleci.com/pipelines/github/bereng/cassandra/854/workflows/d7e989af-0685-4c4c-ae43-951d94e61fe3)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1104869163
##########
src/java/org/apache/cassandra/db/rows/NativeCell.java:
##########
@@ -120,9 +120,9 @@ public int ttl()
return MemoryUtil.getInt(peer + TTL);
}
- public int localDeletionTime()
+ public long localDeletionTime()
{
- return MemoryUtil.getInt(peer + DELETION);
+ return Cell.deletionTimeGuavaUintToLong(MemoryUtil.getInt(peer + DELETION));
Review Comment:
```suggestion
return deletionTimeGuavaUintToLong(MemoryUtil.getInt(peer + DELETION));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1120159590
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,17 +41,31 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ protected final int localDeletionTimeGuavaUint;
+
+ public static DeletionTime getInstance(long markedForDeleteAt, long localDeletionTime)
Review Comment:
We frequently use the name `getInstance` to either access singletons or retrieve cached instances (`MapType`, `ListType`, `CompositeType`, `ToJsonFct`, etc.). However this factory method always returns a new instance, so maybe we could name it `build` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121717326
##########
src/java/org/apache/cassandra/utils/CassandraUInt.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.utils;
+
+import com.google.common.primitives.UnsignedInteger;
+import com.google.common.primitives.UnsignedInts;
+
+/**
+ * This class packages several UInt utility methods so we can easily change it's implementation when needed.
+ */
+public class CassandraUInt
+{
+
+ private static long MAX_VALUE_LONG = UnsignedInteger.MAX_VALUE.longValue();
+ private static int MAX_VALUE_UINT = UnsignedInteger.MAX_VALUE.intValue();
+
+ /**
+ * Converts a long to int's unsigned integer representation
+ *
+ * @param value A long between 0 and 2<sup>32</sup>-1 inclusive
+ * @return an unsigned integer representation of the long
+ * @throws IllegalArgumentException if value '< 0' or '>= 2<sup>32</sup>'
+ */
+ public static int fromLong(long value)
+ {
+ return UnsignedInts.checkedCast(value);
+ }
+
+ /**
+ * Returns the long resulting from parsing the int as an unsigned integer
+ *
+ * @param value Unsigned integer representation of the long
+ * @return The long resulting from parsing the int as an unsigned integer
+ */
+ public static long toLong(int value)
+ {
+ return UnsignedInts.toLong(value);
+ }
+
+ /**
+ * Returns the long representation of the maximum value we can hold
+ */
+ public static long maxValueAsLong()
+ {
+ return MAX_VALUE_LONG;
+ }
+
+ /**
+ * Returns the UInt representation of the maximum value we can hold
+ */
+ public static int maxValueAsUInt()
+ {
+ return MAX_VALUE_UINT;
+ }
Review Comment:
Yeah, I see the point, although I can't imagine an alternative implementation that doesn't cache those commonly used values into constants. Not feeling strong about it either, but if we keep the getters we should probably make the constants `private`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121382070
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +244,29 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ public InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
+ {
+ // We're calling the super constructor with int ldt to force invalid values through
+ // and workaround any validation
+ super(markedForDeleteAt, localDeletionTime);
+ }
+
+ public long localDeletionTime()
Review Comment:
Done
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +244,29 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ public InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
+ {
+ // We're calling the super constructor with int ldt to force invalid values through
+ // and workaround any validation
+ super(markedForDeleteAt, localDeletionTime);
+ }
+
+ public long localDeletionTime()
+ {
+ return localDeletionTimeGuavaUint;
+ }
+
+ public boolean validate()
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122236865
##########
test/unit/org/apache/cassandra/utils/CassandraGenerators.java:
##########
@@ -99,7 +100,7 @@
private static final Gen<SinglePartitionReadCommand> SINGLE_PARTITION_READ_COMMAND_GEN = gen(rnd -> {
TableMetadata metadata = TABLE_METADATA_GEN.generate(rnd);
- int nowInSec = (int) rnd.next(Constraint.between(1, Integer.MAX_VALUE));
+ long nowInSec = rnd.next(Constraint.between(1, Cell.MAX_DELETION_TIME));
Review Comment:
Why `Cell.MAX_DELETION_TIME`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106935056
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
Yep, and we should remove its usage in all places and replace it with a suitable static method that performs whatever sanity checks we want without boxing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108217722
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -630,7 +640,10 @@ public void encode(QueryOptions options, ByteBuf dest, ProtocolVersion version)
if (flags.contains(Flag.KEYSPACE))
CBUtil.writeAsciiString(options.getSpecificOptions().keyspace, dest);
if (flags.contains(Flag.NOW_IN_SECONDS))
- dest.writeInt(options.getSpecificOptions().nowInSeconds);
+ {
+ long nis = options.getSpecificOptions().nowInSeconds == UNSET_NOWINSEC ? UNSET_NOWINSEC_GUAVA_UINT : options.getSpecificOptions().nowInSeconds;
+ dest.writeInt(UnsignedInteger.valueOf(nis).intValue());
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105695597
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -152,7 +156,7 @@ public boolean deletes(long timestamp)
public int dataSize()
{
- return 12;
+ return 16;
Review Comment:
`serializedSize()` in the serializers does indeed account for an int which is what gets written. This `dataSize()` is probably tied to the fact of holding ldt in a long which leads us to your other comment? I'll answer [there](https://github.com/apache/cassandra/pull/1891#pullrequestreview-1297370063)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107057240
##########
test/unit/org/apache/cassandra/cql3/CustomNowInSecondsTest.java:
##########
@@ -73,6 +73,32 @@ private void testSelectQuery(boolean prepared)
assertEquals(0, executeSelect(format("SELECT * FROM %s.%s", ks, tbl), now + day + 1, prepared).size());
}
+ @Test
+ public void testSelectQueryWithOverflow()
+ {
+ testSelectQueryOverflowingIntTimestamps(false);
+ testSelectQueryOverflowingIntTimestamps(true);
+ }
+
+ private void testSelectQueryOverflowingIntTimestamps(boolean prepared)
+ {
+ int ttl = Attributes.MAX_TTL - 10; // Give it a TTL that should overflow 'int' timestamps
+
+ String ks = createKeyspace("CREATE KEYSPACE %s WITH replication={ 'class' : 'SimpleStrategy', 'replication_factor' : 1 }");
+ String tbl = createTable(ks, "CREATE TABLE %s (id int primary key, val int)");
+
+ // insert a row with TTL = 1 day.
Review Comment:
Isn't the TTL 20 years minus ten seconds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108366574
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -94,7 +94,7 @@
* See {@link #testGenerateSstables()} to generate sstables.
* Take care on commit as you need to add the sstable files using {@code git add -f}
*/
- public static final String[] legacyVersions = {"nb", "na", "me", "md", "mc", "mb", "ma"};
+ public static final String[] legacyVersions = {"oa", "nb", "na", "me", "md", "mc", "mb", "ma"};
Review Comment:
Makes sense, it's the `legacyVersions` name what confused me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107047993
##########
test/unit/org/apache/cassandra/db/CellTest.java:
##########
@@ -172,9 +172,6 @@ public void testValidate()
// Invalid ttl
assertInvalid(BufferCell.expiring(c, 0, -4, 4, bbs(4)));
- // Cells with overflowed localExpirationTime are valid after CASSANDRA-14092
Review Comment:
Instead of just removing the tests, we could verify that they are rejected with `value (-1) is outside the range for an unsigned integer value`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108198838
##########
src/java/org/apache/cassandra/net/OutboundConnection.java:
##########
@@ -1127,7 +1127,7 @@ void onCompletedHandshake(Result<MessagingSuccess> result)
FrameEncoder.PayloadAllocator payloadAllocator = success.allocator;
Channel channel = success.channel;
- Established established = new Established(messagingVersion, channel, payloadAllocator, settings);
+ Established established = new Established(success.messagingVersion, channel, payloadAllocator, settings);
Review Comment:
Rolling upgrade tests where the channel version changes were flaky on this. It would only appear after many many repeats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109966985
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,10 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
+ // Guava uint counterpart to the above for serialization purposes. This imposes a year 2186 max value
+ private static final int UNSET_NOWINSEC_GUAVA_UINT = UnsignedInteger.MAX_VALUE.intValue() - 1;
Review Comment:
I don't see what's the utility on `UNSET_NOWINSEC_GUAVA_UINT` here.
That constant is only used on `encode` and `decode` methods when serializing a `long` `nowInSec` as and unsigned integer. However, that serialization/derialization only happens when we have `Flag.NOW_IN_SECONDS`. That flag is only present if the `long` `nowInSec` is distinct to `UNSET_NOWINSEC`.
So IIUC `UNSET_NOWINSEC_GUAVA_UINT` is never used, and it can be removed. Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109578969
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -598,10 +598,12 @@ private static void loadLegacyTable(String tablePattern, String legacyVersion) t
* <p>
* Run this test alone (e.g. from your IDE) when a new version is introduced or format changed
* during development. I.e. remove the {@code @Ignore} annotation temporarily.
+ *
+ * *************RUN THIS TEST ALONE to avoid unwanted compactions, flushes, etc to interfere
Review Comment:
I think adding the comment should do the trick. I would have done it for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109572574
##########
test/unit/org/apache/cassandra/db/NativeCellTest.java:
##########
@@ -125,7 +125,7 @@ private static Cell<?> rndcell(ColumnMetadata col)
{
long timestamp = rand.nextLong();
int ttl = rand.nextInt();
- int localDeletionTime = rand.nextInt();
+ long localDeletionTime = Math.abs(rand.nextInt());
Review Comment:
+1 to the full range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109594423
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,10 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
+ // Guava uint counterpart to the above for serialization purposes. This imposes a year 2186 max value
+ private static final int UNSET_NOWINSEC_GUAVA_UINT = UnsignedInteger.MAX_VALUE.intValue() - 1;
Review Comment:
Well I'll disagree somehow here. Putting some thought into trying to make your design friendlier is a perfectly good reason.Now if your ideas don't get traction then so be it. I don't feel strongly about it. I'll run it through CI next time I commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127764827
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -242,47 +249,52 @@ void mergeNearestPoints()
{
assert isFull() : "DataHolder must be full in order to merge two points";
- final int[] smallestDifference = findPointPairWithSmallestDistance();
+ final long[] smallestDifference = findPointPairWithSmallestDistance();
- final int point1 = smallestDifference[0];
- final int point2 = smallestDifference[1];
+ final long point1 = smallestDifference[0];
+ final long point2 = smallestDifference[1];
- long key = wrap(point1, 0);
- int index = Arrays.binarySearch(data, key);
+ int index = Arrays.binarySearch(points, point1);
if (index < 0)
{
index = -index - 1;
- assert (index < data.length) : "Not found in array";
- assert (unwrapPoint(data[index]) == point1) : "Not found in array";
+ assert (index < points.length) : "Not found in array";
+ assert (points[index] == point1) : "Not found in array";
}
- long value1 = unwrapValue(data[index]);
- long value2 = unwrapValue(data[index + 1]);
+ long value1 = values[index];
+ long value2 = values[index + 1];
- assert (unwrapPoint(data[index + 1]) == point2) : "point2 should follow point1";
+ assert (points[index + 1] == point2) : "point2 should follow point1";
long sum = value1 + value2;
//let's evaluate in long values to handle overflow in multiplication
- int newPoint = saturatingCastToInt((point1 * value1 + point2 * value2) / sum);
+ //long newPoint = saturatingCastToMaxDeletionTime((point1 * value1 + point2 * value2) / sum); // TODO 14227 Better weighted merged point possible?
+ long newPoint = (point2 - point1) >> 1; // approx division by 2 for perf reasons
Review Comment:
How is `(p2-p1)/2` this equivalent to the `(p1*m1+p2*m2)/(p1+p2)` mentioned on the class JavaDoc? I understand the change is aimed at preventing overflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1132010672
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
Will reword comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1479476818
> @adelapena I notice [here](https://github.com/apache/cassandra/commit/7ca806c60a3e080d740fb163c639bb76a520f6ab#diff-7284e8da2bb0ea3e0e965a8f728407b2326f982281d6b37ca0d7cad79039b70eL214), this is Mick's ticket moving trunk to be v5.0, he removes messaging version 41. So I think I have a bug in my PR where I added v41 and v42 tying the DB version bump to a messaging version bump. I should only add one more version correct? This area is a bit fuzzy for me apologies if I am missing sthg
I understand we would need a new `MessagingService.VERSION_50`. I think `VERSION_41` was briefly added to 4.1 but never really used, so it has been removed in 5.0.
However, I see that `VERSION_41` is [still present on `cassandra-4.1`](https://github.com/apache/cassandra/blob/cassandra-4.1.1/src/java/org/apache/cassandra/net/MessagingService.java#L211), and [also in `cassandra-4.1.1`](https://github.com/apache/cassandra/blob/cassandra-4.1.1/src/java/org/apache/cassandra/net/MessagingService.java#L211). @michaelsembwever shouldn't we remove it also from `cassandra-4.1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1111501243
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -42,8 +45,14 @@
public abstract class Cell<V> extends ColumnData
{
public static final int NO_TTL = 0;
- public static final int NO_DELETION_TIME = Integer.MAX_VALUE;
- public static final int MAX_DELETION_TIME = Integer.MAX_VALUE - 1;
+ public static final long NO_DELETION_TIME = Long.MAX_VALUE;
+ // -5 to leave room for any future sentinel values just in case
Review Comment:
No, but loosing 5s vs cornering yourself into a position you'll have no maneuvering room if you need it seems quite beneficial to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1111663766
##########
test/unit/org/apache/cassandra/db/compaction/CompactionsCQLTest.java:
##########
@@ -313,16 +314,18 @@ public void testCompactionInvalidTombstone() throws Throwable
{
DatabaseDescriptor.setCorruptedTombstoneStrategy(Config.CorruptedTombstoneStrategy.exception);
prepare();
+
// write a standard tombstone with negative local deletion time (LDTs are not set by user and should not be negative):
- RowUpdateBuilder rub = new RowUpdateBuilder(getCurrentColumnFamilyStore().metadata(), -1, System.currentTimeMillis() * 1000, 22).clustering(33).delete("b");
- rub.build().apply();
- flush();
- compactAndValidate();
- readAndValidate(true);
- readAndValidate(false);
+ Assertions.assertThatThrownBy(() -> {
+ new RowUpdateBuilder(getCurrentColumnFamilyStore().metadata(),
+ -1,
+ System.currentTimeMillis() * 1000,
+ 22).clustering(33).delete("b");
+ }).isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("is outside the range for an unsigned integer value");
}
- @Test
+ @Test(expected = RuntimeException.class) // msg = "is outside the range for an unsigned integer value"
Review Comment:
This should never happen as negative ldts just can't get in now. The decision in [C14092](https://issues.apache.org/jira/browse/CASSANDRA-14092) was that you should scrub sstables if there were any negative ldts and _not_ support that logic in the storage engine.
So why does this test exist? digging a bit further [C14467](https://issues.apache.org/jira/browse/CASSANDRA-14467) seems to bring in some defensive logic towards negative ldts and other bad scenarios. I can only think this could happen on corrupted sstables which is what c14467 does: mark corrupted sstables. So I need to think if upon _decoding_ (bc we will never encode) a negative ldt outisde the valid uint range we should fail somehow we trigger the c14467 logic, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134431433
##########
src/java/org/apache/cassandra/db/compaction/Scrubber.java:
##########
@@ -383,7 +384,7 @@ private boolean tryAppend(DecoratedKey prevKey, DecoratedKey key, SSTableRewrite
@SuppressWarnings("resource")
private UnfilteredRowIterator getIterator(DecoratedKey key)
{
- RowMergingSSTableIterator rowMergingIterator = new RowMergingSSTableIterator(SSTableIdentityIterator.create(sstable, dataFile, key), outputHandler);
+ RowMergingSSTableIterator rowMergingIterator = new RowMergingSSTableIterator(SSTableIdentityIterator.create(sstable, dataFile, key), outputHandler, sstable.descriptor.version, reinsertOverflowedTTLRows);
Review Comment:
Can we split the long line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135028655
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,17 +42,41 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ protected final int localDeletionTimeUnsignedInteger;
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135031506
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -146,11 +153,16 @@ public void add(RangeTombstone tombstone)
* This method will be faster if the new tombstone sort after all the currently existing ones (this is a common use case),
* but it doesn't assume it.
*/
- public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, int delTime)
+ public void add(ClusteringBound<?> start, ClusteringBound<?> end, long markedAt, long delTime)
Review Comment:
Yeah but it's always been public and I don't want to burden any future callers with the uint conversion burden. I would prefer to leave it?
You also mention it can be private in another comment. I prefer to favor the original implementation, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1469754301
Yep I need to squash, rebase, re-test perf and then I'll ping Benedict for details. I'd prefer to merge and do the downgradability on a separate ticket mainly for my sanity lol, rebase cost and so others might surface any problems the sooner the better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1134053104
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
Great idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1120163138
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +244,29 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ public InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
+ {
+ // We're calling the super constructor with int ldt to force invalid values through
+ // and workaround any validation
+ super(markedForDeleteAt, localDeletionTime);
+ }
+
+ public long localDeletionTime()
Review Comment:
Nit: add `@Override`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122627686
##########
NEWS.txt:
##########
@@ -1,9 +1,9 @@
-PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092)
-------------------------------------------------------------------
+PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092 & CASSANDRA-14227)
+-----------------------------------------------------------------------------------
(General upgrading instructions are available in the next section)
-The maximum expiration timestamp that can be represented by the storage engine is
-2038-01-19T03:14:06+00:00, which means that inserts with TTL thatl expire after
+The maximum expiration timestamp that can be represented by the storage engine has been
+raised to 2106-02-07T06:28:14+00:00, which means that inserts with TTL that expire after
Review Comment:
I don't think we're totally aligned here.
- `-1` is to have room for the special value `LEGACY_OVERFLOWED_DELETION_TIME`
- `-2` is so we don't collide with the above one
- Yes I got 1s wrong good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121955697
##########
NEWS.txt:
##########
@@ -57,6 +57,7 @@ using the provided 'sstableupgrade' tool.
New features
------------
+ - Entended max expiration time from 2038-01-19T03:14:06+00:00 to 2106-02-07T06:28:14+00:00
Review Comment:
Same as before, the current max expiration time is `2106-02-07T06:28:13+00:00`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121381670
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,17 +41,31 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ protected final int localDeletionTimeGuavaUint;
+
+ public static DeletionTime getInstance(long markedForDeleteAt, long localDeletionTime)
Review Comment:
Done
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -187,8 +244,29 @@ public void skip(DataInputPlus in) throws IOException
public long serializedSize(DeletionTime delTime)
{
- return TypeSizes.sizeof(delTime.localDeletionTime())
- + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ return TypeSizes.sizeof(Integer.MAX_VALUE)
+ + TypeSizes.sizeof(delTime.markedForDeleteAt());
+ }
+ }
+
+ // When scrubbing legacy sstables (overflown) or upon sstable corruption we could have negative ldts
+ public static class InvalidDeletionTime extends DeletionTime
+ {
+ public InvalidDeletionTime(long markedForDeleteAt, int localDeletionTime)
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121680912
##########
src/java/org/apache/cassandra/utils/CassandraUInt.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cassandra.utils;
+
+import com.google.common.primitives.UnsignedInteger;
+import com.google.common.primitives.UnsignedInts;
+
+/**
+ * This class packages several UInt utility methods so we can easily change it's implementation when needed.
+ */
+public class CassandraUInt
+{
+
+ private static long MAX_VALUE_LONG = UnsignedInteger.MAX_VALUE.longValue();
+ private static int MAX_VALUE_UINT = UnsignedInteger.MAX_VALUE.intValue();
+
+ /**
+ * Converts a long to int's unsigned integer representation
+ *
+ * @param value A long between 0 and 2<sup>32</sup>-1 inclusive
+ * @return an unsigned integer representation of the long
+ * @throws IllegalArgumentException if value '< 0' or '>= 2<sup>32</sup>'
+ */
+ public static int fromLong(long value)
+ {
+ return UnsignedInts.checkedCast(value);
+ }
+
+ /**
+ * Returns the long resulting from parsing the int as an unsigned integer
+ *
+ * @param value Unsigned integer representation of the long
+ * @return The long resulting from parsing the int as an unsigned integer
+ */
+ public static long toLong(int value)
+ {
+ return UnsignedInts.toLong(value);
+ }
+
+ /**
+ * Returns the long representation of the maximum value we can hold
+ */
+ public static long maxValueAsLong()
+ {
+ return MAX_VALUE_LONG;
+ }
+
+ /**
+ * Returns the UInt representation of the maximum value we can hold
+ */
+ public static int maxValueAsUInt()
+ {
+ return MAX_VALUE_UINT;
+ }
Review Comment:
I think callers can directly access the public `MAX_VALUE_LONG` and `MAX_VALUE_UINT` constants, making these two getters unnecessary. The nice JavaDoc can be moved to the constants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122195723
##########
src/java/org/apache/cassandra/io/sstable/IndexInfo.java:
##########
@@ -47,7 +47,7 @@
* (long) IndexInfo.offset
* (long) IndexInfo.width
* (bool) IndexInfo.endOpenMarker != null (if 3.0)
- * (int) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)
+ * (long) IndexInfo.endOpenMarker.localDeletionTime (if 3.0 && IndexInfo.endOpenMarker != null)
Review Comment:
This refers to serialization, and `DeletionTime#localDeletionTime` is still being serialized as a `int`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122659066
##########
src/java/org/apache/cassandra/db/RangeTombstoneList.java:
##########
@@ -58,12 +58,12 @@ public class RangeTombstoneList implements Iterable<RangeTombstone>, IMeasurable
private ClusteringBound<?>[] starts;
private ClusteringBound<?>[] ends;
private long[] markedAts;
- private int[] delTimes;
+ private long[] delTimes;
Review Comment:
Yep the same can be said about every `long` TS in the codebase. But the underlying idea in the PR is to have a `long` codebase where we don't burden devs with dealing with units leaving that as a 'serialization' implementation detail. Also to not spread uints across the codebase like wildfire. `Cell` and `DeletionTime` being the exceptions.
At the end of the day we're trading a few bytes vs bit-wise operations. Being range tombstones... finger in the air uneducated guess... it might not be worth it but happy to be corrected :shrug: Like what's the usage here? Lot's of operations with many overlaps or many non-overlapping ranges with very little operations?...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105731270
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
It really shouldn't. The cast is free, you just load into a suitably sized register. The truncation will have a single cycle latency (and greater than one-per-cycle throughput). If you can measure the impact of this in a real C* trace, I will eat my hat. If you can even get it to show up in a microbenchmark that does _anything_ real (compaction, memtable read or anything really) I will be very impressed. It's going to be 100% swallowed up by memory latency, data dependencies in the pipeline, etc. You only worry about this kind of thing if you can get far better instructions-per-cycle than we can. This is the lowest of low level optimisation concern.
However, the wasted memory will definitely have an impact. For many JVMs this will require an extra 8-bytes of memory.
So this is a pretty simple tradeoff, and I would not personally waste my time benchmarking it. Software engineering is about picking your battles and where your time is spent, and this is a bad battle to pick! But YMMV of course.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185717994
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,8 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+ USE_OA_SSTABLE_FORMAT("cassandra.use_oa_sstable_format", BigFormat.TTL_MODE.LEGACY_TTL_2038.toString())
Review Comment:
Whatever the new downgradability thing comes it will use whatever mechanics it decides. I still think it won't be a boolean mainly bc during mixed clusters you need a way to dynamically check the cluster status. That check could be in the hot path, such as it's here, so you need a 3rd value to signal the cluster is fully migrated and we can spare that check.
I can add the comment. But I wouldn't mark it as deprecated until that ticket is complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1185976773
##########
test/unit/org/apache/cassandra/cql3/EmptyValuesTest.java:
##########
@@ -76,6 +78,11 @@ private void verify(String emptyValue) throws Throwable
try (PrintStream out = new PrintStream(buf, true))
{
ProcessBuilder pb = new ProcessBuilder("tools/bin/sstabledump", ssTable.getFilename());
+ Map<String, String> envs = pb.environment();
+ envs.put("JVM_OPTS", "-D"
+ + CassandraRelevantProperties.SSTABLE_FORMAT_TTL_MODE.getKey()
+ + "="
+ + CassandraRelevantProperties.SSTABLE_FORMAT_TTL_MODE.getString());
Review Comment:
Without it sstabledump wouldn't be called with the right feature flag value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1203519263
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
Ok notice this ones get executed at the same time as `j8_unit_tests`. I don't require a separate approval for them as I don't think it makes sense to ever avoid them given the codebase should be able to do nc and oa. Makes sense?
This is why I named them after the 'parent' junit job as they sit side by side rather than with the shortened form. I can use underscore though. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210259901
##########
conf/cassandra.yaml:
##########
@@ -1921,3 +1921,30 @@ sstable_formats:
parameters:
id: 0
name: big
+
+
+# This property indicates with what Cassandra major version the storage format will be compatible with.
+#
+# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
+#
+# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
+# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,
+# so that feature won't be available if this property is set to CASSANDRA_4. See upgrade guides for details.
+#
+# Possible values are in the StorageCompatibilityMode.java file accessible online. At the time of writing these are:
+# - CASSANDRA_Y: Stays compatible with the Y.x line in features, formats and component versions
Review Comment:
I will use your latest suggestion in addition to my sentence. I think _Y invites to guessing which we don't want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210388816
##########
conf/cassandra.yaml:
##########
@@ -1921,3 +1921,30 @@ sstable_formats:
parameters:
id: 0
name: big
+
+
+# This property indicates with what Cassandra major version the storage format will be compatible with.
+#
+# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
+#
+# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
+# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,
+# so that feature won't be available if this property is set to CASSANDRA_4. See upgrade guides for details.
+#
+# Possible values are in the StorageCompatibilityMode.java file accessible online. At the time of writing these are:
+# - CASSANDRA_Y: Stays compatible with the Y.x line in features, formats and component versions
Review Comment:
Done
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127873045
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
@adelapena are we good here to resolve?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106930320
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
Agree, `UnsignedInteger` provides validation for the long -> uint conversion, but not for the uint -> long conversion discussed here. `UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue()` doesn't do any validation, and it's equivalent to `UnsignedInts.toLong(deletionTimeGuavaUint)` or `deletionTimeGuavaUint & 0xffffffffL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107077336
##########
src/java/org/apache/cassandra/net/MessagingService.java:
##########
@@ -237,7 +238,9 @@ public static int getVersionOrdinal(int version)
{
VERSION_30(10),
VERSION_3014(11),
- VERSION_40(12);
+ VERSION_40(12),
+ VERSION_41(13),
+ VERSION_42(14);;
Review Comment:
Nit: Double semicolon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105791366
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
Which aligns to my observations where the places I did put that optimization I saw no impact. As I will anyway perf test again before a merge happens then let's just change it +1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106071105
##########
src/java/org/apache/cassandra/db/rows/ArrayCell.java:
##########
@@ -83,23 +86,23 @@ public CellPath path()
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
- return new ArrayCell(newColumn, timestamp, ttl, localDeletionTime, value, path);
+ return new ArrayCell(newColumn, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
}
public Cell<?> withUpdatedValue(ByteBuffer newValue)
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, ByteBufferUtil.getArray(newValue), path);
+ return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), ByteBufferUtil.getArray(newValue), path);
}
- public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, int newLocalDeletionTime)
+ public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, long newLocalDeletionTime)
{
return new ArrayCell(column, newTimestamp, ttl, newLocalDeletionTime, value, path);
}
@Override
public Cell<?> withSkippedValue()
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, EMPTY_BYTE_ARRAY, path);
+ return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), EMPTY_BYTE_ARRAY, path);
Review Comment:
Actually, the three with* methods that call `localDeletionTime` could benefit from a private constructor that takes the unsigned int, so we don't need to transform the deletion time twice:
```java
public ArrayCell(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, byte[] value, CellPath path)
{
this(column, timestamp, ttl, deletionTimeLongToGuavaUint(localDeletionTime), value, path);
}
private ArrayCell(ColumnMetadata column, long timestamp, int ttl, int localDeletionTimeGuavaUint, byte[] value, CellPath path)
{
super(column);
this.timestamp = timestamp;
this.ttl = ttl;
this.localDeletionTimeGuavaUint = localDeletionTimeGuavaUint;
this.value = value;
this.path = path;
}
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
return new ArrayCell(newColumn, timestamp, ttl, localDeletionTimeGuavaUint, value, path);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106128052
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -108,7 +127,7 @@ public ByteBuffer buffer()
* @return the cell local deletion time, or {@code NO_DELETION_TIME} if the cell is neither
* a tombstone nor an expiring one.
*/
- public abstract int localDeletionTime();
+ public abstract long localDeletionTime();
Review Comment:
Every implementation of this method contains a call to `Cell.deletionTimeGuavaUintToLong`. Instead, we could have an additional `protected` method that returns the unsigned int, and put the call to `Cell.deletionTimeGuavaUintToLong` here:
```java
public long localDeletionTime()
{
return deletionTimeGuavaUintToLong(localDeletionTimeAsUnsignedInt());
}
protected abstract int localDeletionTimeAsUnsignedInt();
```
This simplifies a bit the implementations and might skip a conversion on the first constructor of `NativeCell`:
```java
public NativeCell(NativeAllocator allocator,
OpOrder.Group writeOp,
Cell<?> cell)
{
this(allocator,
writeOp,
cell.column(),
cell.timestamp(),
cell.ttl(),
cell.localDeletionTimeAsUnsignedInt(),
cell.buffer(),
cell.path());
}
public NativeCell(NativeAllocator allocator,
OpOrder.Group writeOp,
ColumnMetadata column,
long timestamp,
int ttl,
long localDeletionTime,
ByteBuffer value,
CellPath path)
{
this(allocator, writeOp, column, timestamp, ttl, deletionTimeLongToGuavaUint(lo
}
private NativeCell(NativeAllocator allocator,
OpOrder.Group writeOp,
ColumnMetadata column,
long timestamp,
int ttl,
int localDeletionTimeUint,
ByteBuffer value,
CellPath path)
{
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106947399
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -108,7 +127,7 @@ public ByteBuffer buffer()
* @return the cell local deletion time, or {@code NO_DELETION_TIME} if the cell is neither
* a tombstone nor an expiring one.
*/
- public abstract int localDeletionTime();
+ public abstract long localDeletionTime();
Review Comment:
All the implementations of `Cell#localDeletionTime` are wrapped in a call to `Cell.deletionTimeGuavaUintToLong`:
* ArrayCell: `return deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint);`
* BufferCell: `return deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint);`
* NativeCell: `return deletionTimeGuavaUintToLong(MemoryUtil.getInt(peer + DELETION));`
If we put that call in a common implementation of `Cell#localDeletionTime` and add an abstract `localDeletionTimeAsUnsignedInt ` method, the implementations would be simplified to:
* ArrayCell: `return localDeletionTimeGuavaUint;`
* BufferCell: `return localDeletionTimeGuavaUint;`
* NativeCell: `return MemoryUtil.getInt(peer + DELETION);`
More importantly, we could use that `Cell#localDeletionTimeAsUnsignedInt` method to avoid pointless conversions in `NativeCell(NativeAllocator, OpOrder.Group, Cell<?>)`. At the moment that method calls `Cell#localDeletionTime`, which does a uint->long conversion. That long is passed to the next constructor, that immediately does the reverse long->uint conversion. We can avoid that with the `localDeletionTimeAsUnsignedInt` method shown in the code snippet shown in my previous comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108173461
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -598,10 +598,12 @@ private static void loadLegacyTable(String tablePattern, String legacyVersion) t
* <p>
* Run this test alone (e.g. from your IDE) when a new version is introduced or format changed
* during development. I.e. remove the {@code @Ignore} annotation temporarily.
+ *
+ * *************RUN THIS TEST ALONE to avoid unwanted compactions, flushes, etc to interfere
Review Comment:
I have torn thoughts here: being in the same class sort of makes sense so you don't forget about it. But it's too easy to mess things around as this _has_ to be ran alone :shrug: I hope my comment is good enough to avoid the trouble I ran into to the next developer here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107050283
##########
test/unit/org/apache/cassandra/cql3/validation/miscellaneous/SSTableMetadataTrackingTest.java:
##########
@@ -168,8 +168,8 @@ public void testTrackMetadata_rowMarkerDelete() throws Throwable
assertEquals(metadata.maxTimestamp, metadata2.maxTimestamp);
}
- private static int nowInSec()
+ private static long nowInSec()
{
- return (int) (System.currentTimeMillis() / 1000);
+ return System.currentTimeMillis() / 1000;
Review Comment:
We can use `FBUtilities.nowInSeconds`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109538909
##########
src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java:
##########
@@ -267,7 +271,24 @@ public int serializedSize(Version version, StatsMetadata component) throws IOExc
size += EstimatedHistogram.serializer.serializedSize(component.estimatedPartitionSize);
size += EstimatedHistogram.serializer.serializedSize(component.estimatedCellPerPartitionCount);
size += CommitLogPosition.serializer.serializedSize(component.commitLogIntervals.upperBound().orElse(CommitLogPosition.NONE));
- size += 8 + 8 + 4 + 4 + 4 + 4 + 8 + 8; // mix/max timestamp(long), min/maxLocalDeletionTime(int), min/max TTL, compressionRatio(double), repairedAt (long)
+ size += 8 + 8; // mix/max timestamp(long)
+ if (version.hasLongLocalDeletionTime())
Review Comment:
I'll resolve but feel free to reopen if you disagree ofc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108369118
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -94,7 +94,7 @@
* See {@link #testGenerateSstables()} to generate sstables.
* Take care on commit as you need to add the sstable files using {@code git add -f}
*/
- public static final String[] legacyVersions = {"nb", "na", "me", "md", "mc", "mb", "ma"};
+ public static final String[] legacyVersions = {"oa", "nb", "na", "me", "md", "mc", "mb", "ma"};
Review Comment:
In case you missed my comment in Jira, this patch should not introduce a new major sstable version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127766272
##########
src/java/org/apache/cassandra/repair/messages/ValidationRequest.java:
##########
@@ -61,27 +63,29 @@ public boolean equals(Object o)
@Override
public int hashCode()
{
- return nowInSec;
+ // TODO 14227 better hash possible?
Review Comment:
Doesn't seem worse than the previous hash
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127768908
##########
src/java/org/apache/cassandra/repair/consistent/LocalSession.java:
##########
@@ -72,9 +72,10 @@ public boolean equals(Object o)
public int hashCode()
{
+ // TODO 14227 better has code possible?
Review Comment:
Maybe just `Objects.hash(super.hashCode(), startedAt, lastUpdate);`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1128080125
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
Ah, I see, makes sense. For more clarity, we could add to `CompactionsCQLTest#testCompactionInvalidTombstone` and `CompactionsCQLTest#testCompactionInvalidPartitionDeletion` a comment saying `To generate the sstables with corrupted data, run this commented code in some pre-c14227 branch that allows for negative ldts`.
By the way, I see that the table we are reading, `negative-ldts-invalid-deletions-test/invalidtombstones` has more two more rows than what is written by the commented code, so it seems they haven't been generated only with it, have they? The same happens with `negative-ldts-invalid-deletions-test//invalid_partition_deletion`. Both tables contain the rows `(0, 0, 0), (1, 1, 1)` in addition to the tombstone mentioned in the commented code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108356580
##########
test/unit/org/apache/cassandra/db/CellTest.java:
##########
@@ -416,20 +416,32 @@ private Cell<?> regular(TableMetadata cfm, String columnName, String value, long
return BufferCell.live(cdef, timestamp, ByteBufferUtil.bytes(value));
}
- private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, int localExpirationTime)
+ private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, long localExpirationTime)
{
return expiring(cfm, columnName, value, timestamp, 1, localExpirationTime);
}
- private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, int ttl, int localExpirationTime)
+ private Cell<?> expiring(TableMetadata cfm, String columnName, String value, long timestamp, int ttl, long localExpirationTime)
{
ColumnMetadata cdef = cfm.getColumn(ByteBufferUtil.bytes(columnName));
return new BufferCell(cdef, timestamp, ttl, localExpirationTime, ByteBufferUtil.bytes(value), null);
}
- private Cell<?> deleted(TableMetadata cfm, String columnName, int localDeletionTime, long timestamp)
+ private Cell<?> deleted(TableMetadata cfm, String columnName, long localDeletionTime, long timestamp)
{
ColumnMetadata cdef = cfm.getColumn(ByteBufferUtil.bytes(columnName));
return BufferCell.tombstone(cdef, timestamp, localDeletionTime);
}
+
+ private static void assertThrowsOnInvalidDeletionTime(CheckedFunction runnable)
+ {
+ try
+ {
+ runnable.apply();
+ fail("Expected Exception was not thrown");
+ } catch (IllegalArgumentException e)
Review Comment:
```suggestion
}
catch (IllegalArgumentException e)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1129297958
##########
test/unit/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilderTest.java:
##########
@@ -168,8 +167,8 @@ public void testRounding() throws Exception
builder.update(samples[i]);
TombstoneHistogram hist = builder.build();
assertEquals(hist.size(), 5);
- assertEquals(asMap(hist).get(60).intValue(), 2);
- assertEquals(asMap(hist).get(120).intValue(), 1);
+ assertEquals(asMap(hist).get(60l).intValue(), 2);
+ assertEquals(asMap(hist).get(120l).intValue(), 1);
Review Comment:
Nit: same as before
```suggestion
assertEquals(asMap(hist).get(60L).intValue(), 2);
assertEquals(asMap(hist).get(120L).intValue(), 1);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1542064156
Hi @belliottsmith ,
yes a yaml property was my first thought. But we have lots of statics and statics depending on other statics. When I tried that approach I quickly got into statics hell on init hence the current approach which was easier. In any case I think it's quite flexible as it stands that come the time sbdy wanted to change it it doesn't impose much.
The UPGRADING enum value we have to keep it imo. Here we have to check other node's versions to know how we should behave i.e.. That has a cost, it might even be in a hot path, and future features might rely on CQL, TCM or other costly checks. We want to tell the node he's out of that state and to avoid all that extra load. Also you want to stabilize the cluster in the upgraded state, where accidentally starting a node in an older version i.e., won't make the cluster flap between old/new behaviors accidentally. So yes I agree it'd be prettier and easy to reason with only 2 states but the 3rd one brings good advantages imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1171496735
##########
test/unit/org/apache/cassandra/cql3/CustomNowInSecondsTest.java:
##########
@@ -73,6 +79,48 @@ private void testSelectQuery(boolean prepared)
assertEquals(0, executeSelect(format("SELECT * FROM %s.%s", ks, tbl), now + day + 1, prepared).size());
}
+ @Test
+ public void testSelectQueryWitInthOverflow()
+ {
+ testSelectQueryOverflowingIntTimestamps(false);
+ testSelectQueryOverflowingIntTimestamps(true);
+ }
+
+ private void testSelectQueryOverflowingIntTimestamps(boolean prepared)
Review Comment:
These two methods could have the same name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122188991
##########
src/java/org/apache/cassandra/db/rows/UnfilteredSerializer.java:
##########
@@ -587,7 +588,10 @@ public Row deserializeRowBody(DataInputPlus in,
{
long timestamp = header.readTimestamp(in);
int ttl = hasTTL ? header.readTTL(in) : LivenessInfo.NO_TTL;
- int localDeletionTime = hasTTL ? header.readLocalDeletionTime(in) : LivenessInfo.NO_EXPIRATION_TIME;
+ long localDeletionTime = hasTTL ? header.readLocalDeletionTime(in) : LivenessInfo.NO_EXPIRATION_TIME;
+ if (localDeletionTime < 0)
+ localDeletionTime = helper.version < MessagingService.VERSION_42 ? Cell.LEGACY_OVERFLOWED_DELETION_TIME : Cell.deletionTimeUnsignedIntegerToLong((int) localDeletionTime);
Review Comment:
Nit: split the long line
```suggestion
localDeletionTime = helper.version < MessagingService.VERSION_42
? Cell.LEGACY_OVERFLOWED_DELETION_TIME
: Cell.deletionTimeUnsignedIntegerToLong((int) localDeletionTime);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1118623853
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
Added in the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1120586043
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +69,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? NO_DELETION_TIME_GUAVA_UINT : UnsignedInts.checkedCast(deletionTime);
Review Comment:
Directly using `UnsignedInts` instead of instantiating `UnsignedInteger` is nice. The `UnsignedInts` class is marked as beta and it shows a distracting warning on the four classes where it's used. The warning seems quite ignorable since the two only methods we use look quite simple and safe.
Perhaps we could avoid the warning by wrapping those two methods into our utility class. That should hide the warning and isolate the Guava dependency. It would also make easy to change the implementation in the future, if we wanted to do so. That wrapping would look like:
```java
package org.apache.cassandra.utils;
import com.google.common.primitives.UnsignedInts;
public class UnsignedIntegers
{
public static int fromLong(long value)
{
return UnsignedInts.checkedCast(value);
}
public static long toLong(int value) {
return UnsignedInts.toLong(value);
}
}
```
Since we would be encapsulating the Guava dependency into the utility class, we could rename some vars to make them independent of the implementation. For example, `deletionTimeGuavaUint` could be named just `deletionTimeUint`, or perhaps `deletionTimeUnsignedInteger`.
Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1120586043
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +69,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? NO_DELETION_TIME_GUAVA_UINT : UnsignedInts.checkedCast(deletionTime);
Review Comment:
Directly using `UnsignedInts` instead of instantiating `UnsignedInteger` is nice. The `UnsignedInts` class is marked as beta and it shows a distracting warning on the four classes where it's used. The warning seems quite ignorable since the two only methods we use look quite simple and safe.
Perhaps we could avoid the warning by wrapping those two methods into our own utility class. That should hide the warning and isolate the Guava dependency. It would also make easy to change the implementation in the future, if we wanted to do so. That wrapping would look like:
```java
package org.apache.cassandra.utils;
import com.google.common.primitives.UnsignedInts;
public class UnsignedIntegers
{
public static int fromLong(long value)
{
return UnsignedInts.checkedCast(value);
}
public static long toLong(int value)
{
return UnsignedInts.toLong(value);
}
}
```
Since we would be encapsulating the Guava dependency into the utility class, we could rename some vars to make them independent of the implementation. For example, `deletionTimeGuavaUint` could be named just `deletionTimeUint`, or perhaps `deletionTimeUnsignedInteger`.
Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122619364
##########
test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java:
##########
@@ -637,18 +637,18 @@ public void testGenerateSstables() throws Throwable
File ksDir = new File(LEGACY_SSTABLE_ROOT, String.format("%s/legacy_tables", BigFormat.latestVersion));
ksDir.tryCreateDirectories();
- copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir);
- copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir);
+ copySstablesFromTestData(String.format("legacy_%s_simple", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_simple_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust", BigFormat.latestVersion), ksDir, "legacy_tables");
+ copySstablesFromTestData(String.format("legacy_%s_clust_counter", BigFormat.latestVersion), ksDir, "legacy_tables");
}
- public static void copySstablesFromTestData(String table, File ksDir) throws IOException
+ public static void copySstablesFromTestData(String table, File ksDir, String ks) throws IOException
Review Comment:
Sorry not following you now. We shouldn't need any further changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1450050986
I also prefer to keep the conversations open until the reviewer has acknowledged the changes, or at least remained silent for a time. This makes it easy to follow the discussions on the PR and verify if/how the suggestions have been addressed. I also prefer to follow discussions on the PR page over email notification threads because it groups discussions by topic and code context, instead of just chronologically. I also find difficult to distinguish between open and resolved conversations on email notification threads.
I also think that keeping the conversations open makes them more visible for additional reviewers.
The exception to this are trivial suggestions that are addressed exactly as suggested, such adding an `@Override` annotation, fixing typos, and that kind of nits. IMO those conversations are unlikely to be discussed by the reviewer and thus they can be immediately resolved.
So I guess the rule of thumb would be keeping the conversation open if you think the reviewers might want to review the changes or they might have an objection to them. wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1121953509
##########
NEWS.txt:
##########
@@ -1,9 +1,9 @@
-PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092)
-------------------------------------------------------------------
+PLEASE READ: MAXIMUM TTL EXPIRATION DATE NOTICE (CASSANDRA-14092 & CASSANDRA-14227)
+-----------------------------------------------------------------------------------
(General upgrading instructions are available in the next section)
-The maximum expiration timestamp that can be represented by the storage engine is
-2038-01-19T03:14:06+00:00, which means that inserts with TTL thatl expire after
+The maximum expiration timestamp that can be represented by the storage engine has been
+raised to 2106-02-07T06:28:14+00:00, which means that inserts with TTL that expire after
Review Comment:
This would be the value if `Cell.MAX_DELETION_TIME` were `CassandraUInt.maxValueAsLong() - 1`, which is the maximum that we can represent while leaving `CassandraUInt.maxValueAsLong()` for `Cell.NO_DELETION_TIME`.
However `Cell.MAX_DELETION_TIME` currently is `CassandraUInt.maxValueAsLong() - 2` for leaving space for some unknown future use.
If we are going to keep it that way, the formatted max timestamp mentioned here should be `2106-02-07T06:28:13+00:00`, one second earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1122140400
##########
src/java/org/apache/cassandra/db/ExpirationDateOverflowHandling.java:
##########
@@ -62,23 +62,23 @@
}
public static final String MAXIMUM_EXPIRATION_DATE_EXCEEDED_WARNING = "Request on table {}.{} with {}ttl of {} seconds exceeds maximum supported expiration " +
- "date of 2038-01-19T03:14:06+00:00 and will have its expiration capped to that date. " +
+ "date of 2106-02-07T06:28:14+00:00 and will have its expiration capped to that date. " +
Review Comment:
Same as before, the current max expiration time is `2106-02-07T06:28:13+00:00`, due to `MAX_DELETION_TIME` being `CassandraUInt.maxValueAsLong() - 2` instead of `CassandraUInt.maxValueAsLong() - 1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1135038202
##########
src/java/org/apache/cassandra/db/compaction/Scrubber.java:
##########
@@ -383,7 +384,7 @@ private boolean tryAppend(DecoratedKey prevKey, DecoratedKey key, SSTableRewrite
@SuppressWarnings("resource")
private UnfilteredRowIterator getIterator(DecoratedKey key)
{
- RowMergingSSTableIterator rowMergingIterator = new RowMergingSSTableIterator(SSTableIdentityIterator.create(sstable, dataFile, key), outputHandler);
+ RowMergingSSTableIterator rowMergingIterator = new RowMergingSSTableIterator(SSTableIdentityIterator.create(sstable, dataFile, key), outputHandler, sstable.descriptor.version, reinsertOverflowedTTLRows);
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1171475059
##########
src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java:
##########
@@ -307,8 +310,14 @@ public int serializedSize(Version version, StatsMetadata component) throws IOExc
size += EstimatedHistogram.serializer.serializedSize(component.estimatedPartitionSize);
size += EstimatedHistogram.serializer.serializedSize(component.estimatedCellPerPartitionCount);
size += CommitLogPosition.serializer.serializedSize(component.commitLogIntervals.upperBound().orElse(CommitLogPosition.NONE));
- size += 8 + 8 + 4 + 4 + 4 + 4 + 8 + 8; // mix/max timestamp(long), min/maxLocalDeletionTime(int), min/max TTL, compressionRatio(double), repairedAt (long)
- size += TombstoneHistogram.serializer.serializedSize(component.estimatedTombstoneDropTime);
+ size += 8 + 8; // mix/max timestamp(long)
+ size += 4 + 4; //min/maxLocalDeletionTime(either int or uint)
+ size += 4 + 4 + 8 + 8;// min/max TTL, compressionRatio(double), repairedAt (long)
+ if (version.hasUIntDeletionTime())
+ size += TombstoneHistogram.serializer.serializedSize(component.estimatedTombstoneDropTime);
+ else
+ size += TombstoneHistogram.legacySerializer.serializedSize(component.estimatedTombstoneDropTime);
Review Comment:
There are three similar if-else block on this class. They select the proper serializer according to the sstable version and then both if-else branches perform the same operation for the selected serializer.
I think we could get rid of the repeated if-else blocks if we had a single reusable method to get the serializer for the current sstable version. Also, the serialization tests on `StreamingTombstoneHistogramBuilderTest` should test both serializers. For example, [this way](https://github.com/adelapena/cassandra/commit/13a7a802ebc31142a5221d3f87e39d23b1d651b3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1171610691
##########
conf/jvm-server.options:
##########
@@ -181,8 +181,8 @@
###################################
# Defines how to handle INSERT requests with TTL exceeding the maximum supported expiration date:
-# * REJECT: this is the default policy and will reject any requests with expiration date timestamp after 2038-01-19T03:14:06+00:00.
-# * CAP: any insert with TTL expiring after 2038-01-19T03:14:06+00:00 will expire on 2038-01-19T03:14:06+00:00 and the client will receive a warning.
+# * REJECT: this is the default policy and will reject any requests with expiration date timestamp after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 depending on compatibility mode
+# * CAP: any insert with TTL expiring after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 will expire on 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 and the client will receive a warning.
Review Comment:
We could give some additional details about the two possible max dates, and split the long lines:
```suggestion
# * REJECT: this is the default policy and will reject any requests with expiration date timestamp after
# 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00 depending on compatibility mode (<="nc" sstable formats)
# * CAP: any insert with TTL expiring after 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00, depending on
# compatibility mode (<="nc" sstable formats), will expire on 2106-02-07T06:28:13+00:00 or 2038-01-19T03:14:06+00:00
and the client will receive a warning.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1173377896
##########
src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java:
##########
@@ -307,8 +310,14 @@ public int serializedSize(Version version, StatsMetadata component) throws IOExc
size += EstimatedHistogram.serializer.serializedSize(component.estimatedPartitionSize);
size += EstimatedHistogram.serializer.serializedSize(component.estimatedCellPerPartitionCount);
size += CommitLogPosition.serializer.serializedSize(component.commitLogIntervals.upperBound().orElse(CommitLogPosition.NONE));
- size += 8 + 8 + 4 + 4 + 4 + 4 + 8 + 8; // mix/max timestamp(long), min/maxLocalDeletionTime(int), min/max TTL, compressionRatio(double), repairedAt (long)
- size += TombstoneHistogram.serializer.serializedSize(component.estimatedTombstoneDropTime);
+ size += 8 + 8; // mix/max timestamp(long)
+ size += 4 + 4; //min/maxLocalDeletionTime(either int or uint)
+ size += 4 + 4 + 8 + 8;// min/max TTL, compressionRatio(double), repairedAt (long)
+ if (version.hasUIntDeletionTime())
+ size += TombstoneHistogram.serializer.serializedSize(component.estimatedTombstoneDropTime);
+ else
+ size += TombstoneHistogram.legacySerializer.serializedSize(component.estimatedTombstoneDropTime);
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1173378688
##########
src/java/org/apache/cassandra/io/sstable/format/big/RowIndexEntry.java:
##########
@@ -59,7 +59,7 @@
* (long) position (64 bit long, vint encoded)
* (int) serialized size of data that follows (32 bit int, vint encoded)
* -- following for indexed entries only (so serialized size > 0)
- * (int) DeletionTime.localDeletionTime
+ * (int) int/Uint DeletionTime.localDeletionTime
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1184559832
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -69,11 +73,19 @@ public class BigFormat extends AbstractSSTableFormat<BigTableReader, BigTableWri
{
private final static Logger logger = LoggerFactory.getLogger(BigFormat.class);
+ // Feature flag env variable to use extended TTL up to 2106 with the new oa sstable format. See c14227
+ public static enum TTL_MODE {LEGACY_TTL_2038, COMPATIBILITY, EXTENDED_TTL_2106};
+ @VisibleForTesting
+ public static final String USE_OA_SSTABLE_FORMAT = "cassandra.use_oa_sstable_format";
Review Comment:
I've had these same exact thoughts. But a silent sstable format change doesn't sound pretty to me either. We need to come up with a name/values that link the TTL and sstable concepts imo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107088750
##########
src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java:
##########
@@ -78,7 +79,8 @@ public static SSTableIdentityIterator create(SSTableReader sstable, FileDataInpu
{
dfile.seek(indexEntry.position);
ByteBufferUtil.skipShortLength(dfile); // Skip partition key
- DeletionTime partitionLevelDeletion = DeletionTime.serializer.deserialize(dfile);
+ DeletionTime partitionLevelDeletion;
+ partitionLevelDeletion = DeletionTime.getSerializer(sstable.descriptor.version).deserialize(dfile);
Review Comment:
```suggestion
DeletionTime partitionLevelDeletion = DeletionTime.getSerializer(sstable.descriptor.version).deserialize(dfile);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105653442
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -152,7 +156,7 @@ public boolean deletes(long timestamp)
public int dataSize()
{
- return 12;
+ return 16;
Review Comment:
Since we're writing an int, I don't think this needs to change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1105630097
##########
src/java/org/apache/cassandra/cql3/QueryOptions.java:
##########
@@ -53,6 +54,10 @@
public static final CBCodec<QueryOptions> codec = new Codec();
+ private static final int UNSET_NOWINSEC = Integer.MIN_VALUE;
+ // Guava uint counterpart to the above for serialization purposes. This imposes a year 2186 max value
+ private static final int UNSET_NOWINSEC_GUAVA_UINT = UnsignedInteger.MAX_VALUE.intValue() - 1;
Review Comment:
Why do we subtract one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107277323
##########
src/java/org/apache/cassandra/cql3/Attributes.java:
##########
@@ -128,7 +128,7 @@ public int getTimeToLive(QueryOptions options, TableMetadata metadata) throws In
}
catch (MarshalException e)
{
- throw new InvalidRequestException("Invalid timestamp value: " + tval);
+ throw new InvalidRequestException("Invalid TTL value: " + tval);
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106085273
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
+1 to using `int`, the same way we do it in {{ArrayCell}} and {{BufferCell}}.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106743032
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -59,14 +68,24 @@
public interface Factory<V>
{
- Cell<V> create(ColumnMetadata column, long timestamp, int ttl, int localDeletionTime, V value, CellPath path);
+ Cell<V> create(ColumnMetadata column, long timestamp, int ttl, long localDeletionTime, V value, CellPath path);
}
protected Cell(ColumnMetadata column)
{
super(column);
}
+ public static int deletionTimeLongToGuavaUint(long deletionTime)
+ {
+ return deletionTime == NO_DELETION_TIME ? UnsignedInteger.MAX_VALUE.intValue() : UnsignedInteger.valueOf(deletionTime).intValue();
+ }
+
+ public static long deletionTimeGuavaUintToLong(int deletionTimeGuavaUint)
+ {
+ return deletionTimeGuavaUint == UnsignedInteger.MAX_VALUE.intValue() ? NO_DELETION_TIME : UnsignedInteger.fromIntBits(deletionTimeGuavaUint).longValue();
Review Comment:
`UnsignedInts` is sthg I started with. You can see in the trail of commits it was there. I have a very strong opinionated opposition to using it. This comes from the experience of working in this PR and the diabolical debug sessions I had to deal with.
Think that `UnsignedInts` will just swallow any value you throw at it. It doesn't sanity check anything. The bugs that emerge are just gibberish timestamps that don't _crash_ the code. You just get silent row removal, silent row mismatch, etc. Just think in terms of the code randomly changing timestamps and how this would fan out.
Given the above consider also the future of the codebase. If we sometime build some new code that accidentally produces an invalid value the bugs that will emerge from that will be probably silent and 100% sure hard to find. Think sentinel values, usually 'out of band' values that can easily fall outside a valid uint range...
It would be very painful for me to move away from `UnsignedInteger` knowing what I've learnt, it would be very unwise imo and an ugly accident waiting to happen. I would strongly recommend staying with 'UnsignedInteger'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108269664
##########
test/unit/org/apache/cassandra/db/CellTest.java:
##########
@@ -172,9 +172,6 @@ public void testValidate()
// Invalid ttl
assertInvalid(BufferCell.expiring(c, 0, -4, 4, bbs(4)));
- // Cells with overflowed localExpirationTime are valid after CASSANDRA-14092
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1132006878
##########
test/unit/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilderTest.java:
##########
@@ -124,10 +125,8 @@ public void testNumericTypes() throws Exception
builder.update(2);
builder.update(2, Integer.MAX_VALUE); // To check that value overflow is handled correctly
TombstoneHistogram hist = builder.build();
- Map<Integer, Integer> asMap = asMap(hist);
-
- assertEquals(1, asMap.size());
- assertEquals(Integer.MAX_VALUE, asMap.get(2).intValue());
+ Map<Long, Integer> asMap = asMap(hist);
+ assertEquals(Integer.MAX_VALUE, asMap.get(2l).intValue());
Review Comment:
Ah you see, on mine it reads perfectly well but yes, `L` should be better in other typographies. Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1127740134
##########
src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java:
##########
@@ -88,15 +91,15 @@ public StreamingTombstoneHistogramBuilder(int maxBinSize, int maxSpoolSize, int
*
* @param point the point to be added
*/
- public void update(int point)
+ public void update(long point)
Review Comment:
The class JavaDoc should be updated to reflect the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106056462
##########
src/java/org/apache/cassandra/db/rows/ArrayCell.java:
##########
@@ -83,23 +86,23 @@ public CellPath path()
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
- return new ArrayCell(newColumn, timestamp, ttl, localDeletionTime, value, path);
+ return new ArrayCell(newColumn, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
}
public Cell<?> withUpdatedValue(ByteBuffer newValue)
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, ByteBufferUtil.getArray(newValue), path);
+ return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), ByteBufferUtil.getArray(newValue), path);
Review Comment:
This can just call to `localDeletionTime()`
##########
src/java/org/apache/cassandra/db/rows/ArrayCell.java:
##########
@@ -83,23 +86,23 @@ public CellPath path()
public Cell<?> withUpdatedColumn(ColumnMetadata newColumn)
{
- return new ArrayCell(newColumn, timestamp, ttl, localDeletionTime, value, path);
+ return new ArrayCell(newColumn, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), value, path);
}
public Cell<?> withUpdatedValue(ByteBuffer newValue)
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, ByteBufferUtil.getArray(newValue), path);
+ return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), ByteBufferUtil.getArray(newValue), path);
}
- public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, int newLocalDeletionTime)
+ public Cell<?> withUpdatedTimestampAndLocalDeletionTime(long newTimestamp, long newLocalDeletionTime)
{
return new ArrayCell(column, newTimestamp, ttl, newLocalDeletionTime, value, path);
}
@Override
public Cell<?> withSkippedValue()
{
- return new ArrayCell(column, timestamp, ttl, localDeletionTime, EMPTY_BYTE_ARRAY, path);
+ return new ArrayCell(column, timestamp, ttl, deletionTimeGuavaUintToLong(localDeletionTimeGuavaUint), EMPTY_BYTE_ARRAY, path);
Review Comment:
This can just call to `localDeletionTime()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106960348
##########
src/java/org/apache/cassandra/db/DeletionTime.java:
##########
@@ -38,14 +41,15 @@ public class DeletionTime implements Comparable<DeletionTime>, IMeasurableMemory
/**
* A special DeletionTime that signifies that there is no top-level (row) tombstone.
*/
- public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Integer.MAX_VALUE);
+ public static final DeletionTime LIVE = new DeletionTime(Long.MIN_VALUE, Long.MAX_VALUE);
public static final Serializer serializer = new Serializer();
+ public static final Serializer legacySerializer = new LegacySerializer();
private final long markedForDeleteAt;
- private final int localDeletionTime;
+ private final long localDeletionTime;
Review Comment:
Yep, the problem with `DeletionTime#validate` can be addressed with a class extending `DeletionTime`, probably `InvalidDeletionTime`, or `OverflownDeletionTime`. There could be a factory method such as:
```java
public static DeletionTime DeletionTime(long markedForDeleteAt, long localDeletionTime)
{
return localDeletionTime >= 0
? new DeletionTime(markedForDeleteAt, localDeletionTime)
: new OverflownDeletionTime(markedForDeleteAt, localDeletionTime);
}
```
Then `DeletionTime` and `InvalidDeletionTime` can have implementations of `validate()` that just return `true` or `false`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1106976807
##########
src/java/org/apache/cassandra/db/rows/Cell.java:
##########
@@ -108,7 +127,7 @@ public ByteBuffer buffer()
* @return the cell local deletion time, or {@code NO_DELETION_TIME} if the cell is neither
* a tombstone nor an expiring one.
*/
- public abstract int localDeletionTime();
+ public abstract long localDeletionTime();
Review Comment:
Also, `NativeCell#withUpdatedTimestampAndLocalDeletionTime`, `NativeCell#withUpdatedColumn` and `NativeCell#withSkippedValue` could also benefit from that `Cell#localDeletionTimeAsUnsignedInt` method to avoid the uint->long->uint conversion that the currently do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1107154459
##########
src/java/org/apache/cassandra/db/LivenessInfo.java:
##########
@@ -60,18 +60,33 @@ protected LivenessInfo(long timestamp)
this.timestamp = timestamp;
}
- public static LivenessInfo create(long timestamp, int nowInSec)
+ public static LivenessInfo create(long timestamp, long nowInSec)
{
return new LivenessInfo(timestamp);
}
- public static LivenessInfo expiring(long timestamp, int ttl, int nowInSec)
+ public static LivenessInfo expiring(long timestamp, int ttl, long nowInSec)
{
assert ttl != EXPIRED_LIVENESS_TTL;
return new ExpiringLivenessInfo(timestamp, ttl, ExpirationDateOverflowHandling.computeLocalExpirationTime(nowInSec, ttl));
}
- public static LivenessInfo create(long timestamp, int ttl, int nowInSec)
+ // c14227 do not use. Special use to circumvent the overflow policy when scrubbing
+ public static LivenessInfo expiring(long timestamp, int ttl, long nowInSec, boolean applyOverflowPolicy)
+ {
+ assert ttl != EXPIRED_LIVENESS_TTL;
+ return new ExpiringLivenessInfo(timestamp, ttl, applyOverflowPolicy ? ExpirationDateOverflowHandling.computeLocalExpirationTime(nowInSec, ttl) : nowInSec);
+ }
+
+ // c14227 do not use. Special use to circumvent the overflow policy when scrubbing
+ public static LivenessInfo create(long timestamp, int ttl, long nowInSec, boolean applyOverflowPolicy)
Review Comment:
Can be `private`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] belliottsmith commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1108199066
##########
test/unit/org/apache/cassandra/cql3/validation/entities/SecondaryIndexTest.java:
##########
@@ -1575,7 +1575,7 @@ private void testIndexOnPartitionKeyInsertExpiringColumn(boolean flushBeforeUpda
flush();
execute("UPDATE %s USING TTL 1 SET b = 10 WHERE k1 = 1 AND k2 = 2");
- Thread.sleep(1000);
+ Thread.sleep(1500);
Review Comment:
It would be nice to not rely on fudge factors if we can.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1109597735
##########
src/java/org/apache/cassandra/io/sstable/metadata/StatsMetadata.java:
##########
@@ -267,7 +271,24 @@ public int serializedSize(Version version, StatsMetadata component) throws IOExc
size += EstimatedHistogram.serializer.serializedSize(component.estimatedPartitionSize);
size += EstimatedHistogram.serializer.serializedSize(component.estimatedCellPerPartitionCount);
size += CommitLogPosition.serializer.serializedSize(component.commitLogIntervals.upperBound().orElse(CommitLogPosition.NONE));
- size += 8 + 8 + 4 + 4 + 4 + 4 + 8 + 8; // mix/max timestamp(long), min/maxLocalDeletionTime(int), min/max TTL, compressionRatio(double), repairedAt (long)
+ size += 8 + 8; // mix/max timestamp(long)
+ if (version.hasLongLocalDeletionTime())
Review Comment:
Works for me. By the way, I think reviewers cannot reopen conversations once they are marked as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1203864543
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
> I was going to look into this and then decided everything is too nosy and cluttered already. We should use REPEATED_ANT_TEST_TARGET for all these variations imo.
Do you want to undo the recent work on CircleCI, automatic detection of tests, etc. to go back to the old ant task?
The reason for having multiple repeated tasks is in the pre-commit workflow is automatically detecting new tests and running them with all possible configs (such as with and without `oa`). Consider also that we might need to run different types of tests (utest, jvm-dtest, upgrade) in the same push, and we can't do that with a single `repeated_ant_test` job.
The separate workflow also uses multiple repeated tasks for symmetry and to allow different repeated runs into the same push.
> I would leave it as it is and open a ticket to drop all specialized ant based repeats, use the generic ant repeat instead and document it properly
I celebrate all efforts on improving CI config, but this patch should either follow the current approach at adding new jobs or stay unmerged until those separate improvements are done. The current jobs shouldn't be merged in their current inconsistent state, which misses the automatic repetition of tests with `oa`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1210248491
##########
conf/cassandra.yaml:
##########
@@ -1921,3 +1921,30 @@ sstable_formats:
parameters:
id: 0
name: big
+
+
+# This property indicates with what Cassandra major version the storage format will be compatible with.
+#
+# The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+# etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+# Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+# or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
+#
+# This will also determine if certain features depending on newer formats are available. For example, extended TTLs
+# up to 2106 depend on the sstable, commitlog, hints and messaging versions that were introduced by Cassandra 5.0,
+# so that feature won't be available if this property is set to CASSANDRA_4. See upgrade guides for details.
+#
+# Possible values are in the StorageCompatibilityMode.java file accessible online. At the time of writing these are:
+# - CASSANDRA_Y: Stays compatible with the Y.x line in features, formats and component versions
Review Comment:
Ah, I see. I would have added it to the `CASSANDRA_Y` entry in the paragraph about possible values, like in:
```
# - CASSANDRA_X: Stays compatible with the X.y line in features, formats and component versions. Currently, the only
# supported major in CASSANDRA_4.
```
Although what I meant by explicitly mentioning every supported major, which at the moment is only CASSANDRA_4, was the more straightforward:
```
# - CASSANDRA_4: Stays compatible with the 4.x line in features, formats and component versions.
```
But I don't have strong feelings about this, leave it as you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1209709462
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,15 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+
+ /**
+ * {@link }}StorageCompatibility mode sets how the node will behave, sstable or messaging versions to use etc according to a yaml setting.
+ * But many tests don't load the config hence we need to force it otherwise they would run always under the default.
+ *
+ * This is a dev/CI only property. Do not use otherwise.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "bereng (via GitHub)" <gi...@apache.org>.
bereng commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1184537402
##########
src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java:
##########
@@ -363,7 +375,7 @@ public BigTableWriter.Builder builder(Descriptor descriptor)
// we always incremented the major version.
static class BigVersion extends Version
{
- public static final String current_version = "nc";
+ public static final String current_version = BigFormat.ttlMode.equals(TTL_MODE.LEGACY_TTL_2038) ? "nc": "oa";
Review Comment:
Will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1184944923
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,8 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+ USE_OA_SSTABLE_FORMAT("cassandra.use_oa_sstable_format", BigFormat.TTL_MODE.LEGACY_TTL_2038.toString())
Review Comment:
I understand that this property will go away or change to a boolean as soon as we have the full downgradability thing and/or a way to allow users decide the sstable format. So perhaps we should add a comment about that and mark the property with `@Deprecated`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] smiklosovic closed pull request #1891: Cassandra 14227 trunk
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic closed pull request #1891: Cassandra 14227 trunk
URL: https://github.com/apache/cassandra/pull/1891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] adelapena commented on a diff in pull request #1891: Cassandra 14227 trunk
Posted by "adelapena (via GitHub)" <gi...@apache.org>.
adelapena commented on code in PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#discussion_r1188396225
##########
src/java/org/apache/cassandra/config/CassandraRelevantProperties.java:
##########
@@ -360,6 +361,22 @@
FLUSH_LOCAL_SCHEMA_CHANGES("cassandra.test.flush_local_schema_changes", "true"),
TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS("cassandra.streaminghistogram.roundseconds", "60"),
+
+ /**
+ * This property indicates with what Cassandra major version the storage format will be compatible with.
+ *
+ * The chosen storage compatiblity mode will determine the versions of the written sstables, commitlogs, hints,
+ * etc. Those storage elements will use the higher minor versions of the major version that corresponds to the
+ * Cassandra version we want to stay compatible with. For example, if we want to stay compatible with Cassandra 4.0
+ * or 4.1, the value of this property should be 4, and that will make us use 'nc' sstables.
Review Comment:
I forgot this when changing from int to enum:
```suggestion
* or 4.1, the value of this property should be {@code CASSANDRA_4}, and that will make us use 'nc' sstables.
```
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
All the JUnit-based test jobs have a `_repeat` companion job, such as `j8_unit_tests_repeat`, `j8_utests_cdc_repeat`, etc. We need to add analogous `j8_utests_oa_repeat` and `j11_utests_oa_repeat` jobs.
The script `generate.sh` should also be updated to consider those new jobs.
Please let me know if you need any help with this.
##########
.circleci/config_template.yml:
##########
@@ -257,6 +257,10 @@ j8_separate_jobs: &j8_separate_jobs
requires:
- start_j8_unit_tests
- j8_build
+ - j8_unit_tests-oa:
Review Comment:
Nit: Job names should use underscores as separators, as in `j8_unit_tests_oa` and `j11_unit_tests_oa`.
Also, for brevity and consistency with other specialized unit tests such as `j8_utests_compression`, `j8_utests_cdc` and `j8_utests_trie`, they should probably be named `j8_utests_oa` and `j11_utests_oa`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] michaelsembwever commented on pull request #1891: Cassandra 14227 trunk
Posted by "michaelsembwever (via GitHub)" <gi...@apache.org>.
michaelsembwever commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1479777744
> I think VERSION_41 was briefly added to 4.1 but never really used, so it has been removed in 5.0.
Correct. There was a change in the tests that introduced _41, but it was agreed that _41 could be removed (that we didn't need to provide compatibility between test runs within a development cycle).
> However, I see that VERSION_41 is [still present on cassandra-4.1](https://github.com/apache/cassandra/blob/cassandra-4.1.1/src/java/org/apache/cassandra/net/MessagingService.java#L211), and [also in cassandra-4.1.1](https://github.com/apache/cassandra/blob/cassandra-4.1.1/src/java/org/apache/cassandra/net/MessagingService.java#L211). @michaelsembwever shouldn't we remove it also from cassandra-4.1?
Both links are same. But yes, _41 should be removed from 4.1 too (my bad), you can see (in the same class) that it's not actually used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
[GitHub] [cassandra] bereng commented on pull request #1891: Cassandra 14227 trunk
Posted by GitBox <gi...@apache.org>.
bereng commented on PR #1891:
URL: https://github.com/apache/cassandra/pull/1891#issuecomment-1261877286
CI:
- [Circle](https://app.circleci.com/pipelines/github/bereng/cassandra/764/workflows/6a0bd156-6942-4830-88b5-715d65b9821d)
- [Jenkins](https://ci-cassandra.apache.org/view/all/job/Cassandra-devbranch/1954/) (ongoing)
The rolling upgrade failure is a 10% flaky under investigation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org