You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "belliottsmith (via GitHub)" <gi...@apache.org> on 2025/07/03 09:29:29 UTC

[PR] Debug txn journal [cassandra]

belliottsmith opened a new pull request, #4225:
URL: https://github.com/apache/cassandra/pull/4225

   Thanks for sending a pull request! Here are some tips if you're new here:
    
    * Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
    * Be sure to keep the PR description updated to reflect all changes.
    * Write your PR title to summarize what this PR proposes.
    * If possible, provide a concise example to reproduce the issue for a faster review.
    * Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
    * If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
    
   Commit messages should follow the following format:
   
   ```
   <One sentence description, usually Jira title or CHANGES.txt summary>
   
   <Optional lengthier description (context on patch)>
   
   patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
   
   Co-authored-by: Name1 <email1>
   Co-authored-by: Name2 <email2>
   
   ```
   
   The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2183755657


##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +
+                        "  command_store_id int,\n" +
+                        "  message text,\n" +
+                        "  PRIMARY KEY (txn_id, event_type, id_micros, at_micros)" +
+                        ')', UTF8Type.instance));
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().eraseEvents(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.lowerEndpoint().size() != 1) throw invalidRequest("Deletion restricted by lower bound on id_micros or at_micros is unsupported");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED && range.upperEndpoint().size() == 1)) throw invalidRequest("Range deletion must specify one event_type");
+            if (!range.upperEndpoint().value(0).equals(range.lowerEndpoint().value(0))) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.upperEndpoint().size() > 2) throw invalidRequest("Deletion restricted by upper bound on at_micros is unsupported");
+            TraceEventType eventType = parseEventType(range.lowerEndpoint().value(0));
+            if (range.upperEndpoint().size() == 1)
+            {
+                tracing().eraseEvents(txnId, eventType);
+            }
+            else
+            {
+                long before = range.upperEndpoint().value(1);
+                tracing().eraseEventsBefore(txnId, eventType, before);
+            }
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+
+        @Override
+        public DataSet data()
+        {
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing().forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    e.messages().forEach(m -> {
+                        dataSet.row(txnId.toString(), eventType.name(), e.atMicros, m.atNanos - e.atNanos)
+                               .column("command_store_id", m.commandStoreId)
+                               .column("message", m.message);
+                    });
+                });
+            });
+            return dataSet;
+        }
+    }
+
+    public static final class TxnTable extends AbstractVirtualTable implements AbstractVirtualTable.DataSet
+    {
+        static class Entry
+        {
+            final int commandStoreId;
+            final Command command;
+
+            Entry(int commandStoreId, Command command)
+            {
+                this.commandStoreId = commandStoreId;
+                this.command = command;
+            }
+        }
+        private TxnTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN,
+                        "Accord per-CommandStore Transaction State",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  command_store_id int,\n" +
+                        "  save_status text,\n" +
+                        "  route text,\n" +
+                        "  durability text,\n" +
+                        "  execute_at text,\n" +
+                        "  txn text,\n" +
+                        "  deps text,\n" +
+                        "  waiting_on text,\n" +
+                        "  writes text,\n" +
+                        "  result text,\n" +
+                        "  participants_owns text,\n" +
+                        "  participants_touches text,\n" +
+                        "  participants_has_touched text,\n" +
+                        "  participants_executes text,\n" +
+                        "  participants_waits_on text,\n" +
+                        "  PRIMARY KEY (txn_id, command_store_id)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            return this;
+        }
+
+        @Override
+        public boolean isEmpty()
+        {
+            return false;
+        }
+
+        @Override
+        public Partition getPartition(DecoratedKey partitionKey)
+        {
+            String txnIdStr = UTF8Type.instance.compose(partitionKey.getKey());
+            TxnId txnId = TxnId.parse(txnIdStr);
+
+            List<Entry> commands = new ArrayList<>();
+            AccordService.instance().node().commandStores().forEachCommandStore(store -> {
+                Command command = ((AccordCommandStore)store).loadCommand(txnId);
+                if (command != null)
+                    commands.add(new Entry(store.id(), command));
+            });
+
+            if (commands.isEmpty())
+                return null;
+
+            SimpleDataSet ds = new SimpleDataSet(metadata);
+            for (Entry e : commands)
+            {
+                Command command = e.command;
+                ds.row(txnIdStr, e.commandStoreId)
+                  .column("save_status", Objects.toString(command.saveStatus()))
+                  .column("route", Objects.toString(command.route()))
+                  .column("participants_owns", toStr(command, StoreParticipants::owns, StoreParticipants::stillOwns))
+                  .column("participants_touches", toStr(command, StoreParticipants::touches, StoreParticipants::stillTouches))
+                  .column("participants_has_touched", Objects.toString(command.participants().hasTouched()))
+                  .column("participants_executes", toStr(command, StoreParticipants::executes, StoreParticipants::stillExecutes))
+                  .column("participants_waits_on", toStr(command, StoreParticipants::waitsOn, StoreParticipants::stillWaitsOn))
+                  .column("durability", Objects.toString(command.durability()))
+                  .column("execute_at", Objects.toString(command.executeAt()))
+                  .column("txn", Objects.toString(command.partialTxn()))
+                  .column("deps", Objects.toString(command.partialDeps()))
+                  .column("waiting_on", Objects.toString(command.waitingOn()))
+                  .column("writes", Objects.toString(command.writes()))
+                  .column("result", Objects.toString(command.result()));
+            }
+
+            return ds.getPartition(partitionKey);
+        }
+
+        @Override
+        public Iterator<Partition> getPartitions(DataRange range)
+        {
+            throw new UnsupportedOperationException();
+        }
+    }
+
+    public static final class JournalTable extends AbstractVirtualTable implements AbstractVirtualTable.DataSet
+    {
+        static class Entry
+        {
+            final int commandStoreId;
+            final long segment;
+            final int position;
+            final CommandChange.Builder builder;
+
+            Entry(int commandStoreId, long segment, int position, CommandChange.Builder builder)
+            {
+                this.commandStoreId = commandStoreId;
+                this.segment = segment;
+                this.position = position;
+                this.builder = builder;
+            }
+        }
+
+        private JournalTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, JOURNAL,
+                        "Accord per-CommandStore Journal State",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  command_store_id int,\n" +
+                        "  segment bigint,\n" +
+                        "  segment_position int,\n" +
+                        "  save_status text,\n" +
+                        "  route text,\n" +
+                        "  durability text,\n" +
+                        "  execute_at text,\n" +
+                        "  executes_at_least text,\n" +
+                        "  txn text,\n" +
+                        "  deps text,\n" +

Review Comment:
   waiting on here?



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -95,6 +112,10 @@ public class AccordDebugKeyspace extends VirtualKeyspace
     public static final String REDUNDANT_BEFORE   = "redundant_before";
     public static final String REJECT_BEFORE      = "reject_before";
     public static final String TXN_BLOCKED_BY     = "txn_blocked_by";
+    public static final String TXN                = "txn";
+    public static final String JOURNAL            = "journal";
+    public static final String TXN_TRACE          = "txn_trace";
+    public static final String TXN_TRACES         = "txn_traces";

Review Comment:
   should we rename it to `txn_trace_events`? 



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +

Review Comment:
   nit: could you make order here consistent with primary key order? 



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -126,8 +151,8 @@ private DurabilityServiceTable()
                         "  token_sort blob,\n" +
                         "  token_start text,\n" +
                         "  token_end text,\n" +
-                        "  last_started_at_micros bigint,\n" +
-                        "  cycle_started_at_micros bigint,\n" +
+                        "  last_started_at bigint,\n" +
+                        "  cycle_started_at bigint,\n" +

Review Comment:
   Not this patches' problem but 
   ```
                   result.column("transactional_mode", table.params.transactionalMode.toString());
                   result.column("transactional_migration_from", table.params.transactionalMigrationFrom.toString());
   ```



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -126,8 +151,8 @@ private DurabilityServiceTable()
                         "  token_sort blob,\n" +
                         "  token_start text,\n" +
                         "  token_end text,\n" +
-                        "  last_started_at_micros bigint,\n" +
-                        "  cycle_started_at_micros bigint,\n" +
+                        "  last_started_at bigint,\n" +
+                        "  cycle_started_at bigint,\n" +
                         "  retries int,\n" +

Review Comment:
   not this patches' problem, but retries min and stopped are not populated



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +
+                        "  command_store_id int,\n" +
+                        "  message text,\n" +
+                        "  PRIMARY KEY (txn_id, event_type, id_micros, at_micros)" +
+                        ')', UTF8Type.instance));
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().eraseEvents(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.lowerEndpoint().size() != 1) throw invalidRequest("Deletion restricted by lower bound on id_micros or at_micros is unsupported");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED && range.upperEndpoint().size() == 1)) throw invalidRequest("Range deletion must specify one event_type");
+            if (!range.upperEndpoint().value(0).equals(range.lowerEndpoint().value(0))) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.upperEndpoint().size() > 2) throw invalidRequest("Deletion restricted by upper bound on at_micros is unsupported");
+            TraceEventType eventType = parseEventType(range.lowerEndpoint().value(0));
+            if (range.upperEndpoint().size() == 1)
+            {
+                tracing().eraseEvents(txnId, eventType);
+            }
+            else
+            {
+                long before = range.upperEndpoint().value(1);
+                tracing().eraseEventsBefore(txnId, eventType, before);
+            }
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+
+        @Override
+        public DataSet data()
+        {
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing().forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    e.messages().forEach(m -> {
+                        dataSet.row(txnId.toString(), eventType.name(), e.atMicros, m.atNanos - e.atNanos)
+                               .column("command_store_id", m.commandStoreId)
+                               .column("message", m.message);
+                    });
+                });
+            });
+            return dataSet;
+        }
+    }
+
+    public static final class TxnTable extends AbstractVirtualTable implements AbstractVirtualTable.DataSet
+    {
+        static class Entry
+        {
+            final int commandStoreId;
+            final Command command;
+
+            Entry(int commandStoreId, Command command)
+            {
+                this.commandStoreId = commandStoreId;
+                this.command = command;
+            }
+        }
+        private TxnTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN,
+                        "Accord per-CommandStore Transaction State",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  command_store_id int,\n" +
+                        "  save_status text,\n" +
+                        "  route text,\n" +
+                        "  durability text,\n" +
+                        "  execute_at text,\n" +
+                        "  txn text,\n" +
+                        "  deps text,\n" +
+                        "  waiting_on text,\n" +
+                        "  writes text,\n" +
+                        "  result text,\n" +
+                        "  participants_owns text,\n" +
+                        "  participants_touches text,\n" +
+                        "  participants_has_touched text,\n" +
+                        "  participants_executes text,\n" +
+                        "  participants_waits_on text,\n" +
+                        "  PRIMARY KEY (txn_id, command_store_id)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            return this;
+        }
+
+        @Override
+        public boolean isEmpty()
+        {
+            return false;
+        }
+
+        @Override
+        public Partition getPartition(DecoratedKey partitionKey)
+        {
+            String txnIdStr = UTF8Type.instance.compose(partitionKey.getKey());
+            TxnId txnId = TxnId.parse(txnIdStr);
+
+            List<Entry> commands = new ArrayList<>();
+            AccordService.instance().node().commandStores().forEachCommandStore(store -> {
+                Command command = ((AccordCommandStore)store).loadCommand(txnId);
+                if (command != null)
+                    commands.add(new Entry(store.id(), command));
+            });
+
+            if (commands.isEmpty())
+                return null;
+
+            SimpleDataSet ds = new SimpleDataSet(metadata);
+            for (Entry e : commands)
+            {
+                Command command = e.command;
+                ds.row(txnIdStr, e.commandStoreId)
+                  .column("save_status", Objects.toString(command.saveStatus()))
+                  .column("route", Objects.toString(command.route()))
+                  .column("participants_owns", toStr(command, StoreParticipants::owns, StoreParticipants::stillOwns))
+                  .column("participants_touches", toStr(command, StoreParticipants::touches, StoreParticipants::stillTouches))
+                  .column("participants_has_touched", Objects.toString(command.participants().hasTouched()))
+                  .column("participants_executes", toStr(command, StoreParticipants::executes, StoreParticipants::stillExecutes))
+                  .column("participants_waits_on", toStr(command, StoreParticipants::waitsOn, StoreParticipants::stillWaitsOn))
+                  .column("durability", Objects.toString(command.durability()))
+                  .column("execute_at", Objects.toString(command.executeAt()))

Review Comment:
   Do we want executes_at_least here?



##########
src/java/org/apache/cassandra/service/accord/AccordTracing.java:
##########
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.AbstractList;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Predicate;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.Tracing;
+import accord.api.TraceEventType;
+import accord.local.CommandStore;
+import accord.primitives.TxnId;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.NoSpamLogger;
+
+public class AccordTracing
+{
+    private static final int MAX_EVENTS = 10000;
+    private static final Logger logger = LoggerFactory.getLogger(AccordTracing.class);
+    private static final NoSpamLogger noSpamLogger = NoSpamLogger.getLogger(logger, 1L, TimeUnit.MINUTES);
+
+    public interface ConsumeState
+    {
+        void accept(TxnId txnId, TraceEventType eventType, int permits, List<Event> events);
+    }
+
+    public static class Message
+    {
+        public final long atNanos;
+        public final int commandStoreId;
+        public final String message;
+
+        Message(int commandStoreId, String message, long atLeastNanos)
+        {
+            this.commandStoreId = commandStoreId;
+            this.message = message;
+            this.atNanos = Math.max(atLeastNanos, Clock.Global.nanoTime());
+        }
+
+        @Override
+        public String toString()
+        {
+            return message;
+        }
+    }
+
+    public static class Event implements Tracing, Comparable<Event>
+    {
+        public final long atMicros = uniqueNowMicros();
+        public final long atNanos = Clock.Global.nanoTime();
+        final List<Message> messages = new ArrayList<>();
+
+        @Override
+        public void trace(CommandStore commandStore, String s)
+        {
+            long prevNanos = messages.isEmpty() ? 0 : messages.get(messages.size() - 1).atNanos;
+            int id = commandStore == null ? -1 : commandStore.id();
+            if (s.length() > 1000)
+                s = s.substring(0, 1000);
+            messages.add(new Message(id, s, prevNanos + 1));
+        }
+
+        @Override
+        public int compareTo(Event that)
+        {
+            return Long.compareUnsigned(this.atMicros, that.atMicros);
+        }
+
+        public List<Message> messages()
+        {
+            return Collections.unmodifiableList(messages);
+        }
+    }
+
+    static class TraceState extends AbstractList<Event>
+    {
+        int permits;
+        int size;
+        Event[] events;
+
+        void addInternal(Event event)
+        {
+            if (events == null) events = new Event[10];
+            else if (size == events.length) events = new Event[size * 2];
+            events[size++] = event;
+        }
+
+        void truncate(int eraseBefore)
+        {
+            System.arraycopy(events, eraseBefore, events, 0, size - eraseBefore);
+            Arrays.fill(events, size - eraseBefore, size, null);
+            size -= eraseBefore;
+        }
+
+        @Override
+        public Event get(int index)
+        {
+            return events[index];
+        }
+
+        @Override
+        public int size()
+        {
+            return size;
+        }
+    }
+
+    private static final AtomicLong lastNowMicros = new AtomicLong();
+    private static long uniqueNowMicros()
+    {
+        long nowMicros = Clock.Global.currentTimeMillis() * 1000;
+        while (true)
+        {
+            long last = lastNowMicros.get();
+            if (last >= nowMicros)
+                return lastNowMicros.incrementAndGet();
+            if (lastNowMicros.compareAndSet(last, nowMicros))
+                return nowMicros;
+        }
+    }
+
+    final Map<TxnId, EnumMap<TraceEventType, TraceState>> stateMap = new ConcurrentHashMap<>();
+    final AtomicInteger count = new AtomicInteger();
+
+    public Tracing trace(TxnId txnId, TraceEventType eventType)
+    {
+        if (!stateMap.containsKey(txnId))
+            return null;
+
+        class Register implements BiFunction<TxnId, EnumMap<TraceEventType, TraceState>, EnumMap<TraceEventType, TraceState>>
+        {
+            Event event;
+
+            @Override
+            public EnumMap<TraceEventType, TraceState> apply(TxnId id, EnumMap<TraceEventType, TraceState> cur)
+            {
+                if (cur == null)
+                    return null;
+
+                TraceState curState = cur.get(eventType);
+                if (curState == null || curState.permits == 0)
+                    return cur;
+
+                if (count.incrementAndGet() >= MAX_EVENTS)
+                {
+                    count.decrementAndGet();
+                    noSpamLogger.warn("Too many Accord trace events stored already; delete some to continue tracing");
+                }
+                else
+                {
+                    curState.permits--;
+                    curState.addInternal(event = new Event());
+                }
+                return cur;
+            }
+        }
+        Register register = new Register();
+        stateMap.compute(txnId, register);
+        return register.event;
+    }
+
+    public void setPermits(TxnId txnId, TraceEventType eventType, int newPermits)
+    {
+        stateMap.compute(txnId, (id, cur) -> {
+            if (newPermits != 0)
+            {
+                if (cur == null)
+                    cur = new EnumMap<>(TraceEventType.class);
+                cur.computeIfAbsent(eventType, ignore -> new TraceState()).permits = newPermits;
+            }
+            else if (cur != null)
+            {
+                TraceState curState = cur.get(eventType);
+                if (curState != null)
+                {
+                    if (!curState.isEmpty()) curState.permits = 0;
+                    else
+                    {
+                        cur.remove(eventType);
+                        if (cur.isEmpty())
+                            return null;
+                    }
+                }
+            }
+            return cur;
+        });
+    }
+
+    public void erasePermits(TxnId txnId)
+    {
+        stateMap.compute(txnId, (id, cur) -> {
+            if (cur == null)
+                return null;
+
+            Iterator<TraceState> iter = cur.values().iterator();
+            while (iter.hasNext())
+            {
+                TraceState state = iter.next();
+                state.permits = 0;
+                if (state.isEmpty()) iter.remove();
+            }
+            return cur.isEmpty() ? null : cur;        });

Review Comment:
   nit: formatting



##########
src/java/org/apache/cassandra/service/accord/AccordTracing.java:
##########
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.AbstractList;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Predicate;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.Tracing;
+import accord.api.TraceEventType;
+import accord.local.CommandStore;
+import accord.primitives.TxnId;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.NoSpamLogger;
+
+public class AccordTracing
+{
+    private static final int MAX_EVENTS = 10000;
+    private static final Logger logger = LoggerFactory.getLogger(AccordTracing.class);
+    private static final NoSpamLogger noSpamLogger = NoSpamLogger.getLogger(logger, 1L, TimeUnit.MINUTES);
+
+    public interface ConsumeState
+    {
+        void accept(TxnId txnId, TraceEventType eventType, int permits, List<Event> events);
+    }
+
+    public static class Message
+    {
+        public final long atNanos;
+        public final int commandStoreId;
+        public final String message;
+
+        Message(int commandStoreId, String message, long atLeastNanos)
+        {
+            this.commandStoreId = commandStoreId;
+            this.message = message;
+            this.atNanos = Math.max(atLeastNanos, Clock.Global.nanoTime());
+        }
+
+        @Override
+        public String toString()
+        {
+            return message;
+        }
+    }
+
+    public static class Event implements Tracing, Comparable<Event>
+    {
+        public final long atMicros = uniqueNowMicros();
+        public final long atNanos = Clock.Global.nanoTime();
+        final List<Message> messages = new ArrayList<>();
+
+        @Override
+        public void trace(CommandStore commandStore, String s)
+        {
+            long prevNanos = messages.isEmpty() ? 0 : messages.get(messages.size() - 1).atNanos;

Review Comment:
   should it be `? 0` here? Why not `atNanos`? 



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable

Review Comment:
   Could you add these to a comment? Just to make it easier to discover how to use this table:
   
   ```
   UPDATE system_accord_debug.txn_traces SET permits = ? WHERE txn_id = ? AND event_type = ?
   SELECT * FROM system_accord_debug.txn_traces WHERE txn_id = ? AND event_type = ?
   ```



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +601,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();

Review Comment:
   nit can use `tracing()` instance method here



##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +
+                        "  command_store_id int,\n" +
+                        "  message text,\n" +
+                        "  PRIMARY KEY (txn_id, event_type, id_micros, at_micros)" +
+                        ')', UTF8Type.instance));
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().eraseEvents(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.lowerEndpoint().size() != 1) throw invalidRequest("Deletion restricted by lower bound on id_micros or at_micros is unsupported");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED && range.upperEndpoint().size() == 1)) throw invalidRequest("Range deletion must specify one event_type");
+            if (!range.upperEndpoint().value(0).equals(range.lowerEndpoint().value(0))) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.upperEndpoint().size() > 2) throw invalidRequest("Deletion restricted by upper bound on at_micros is unsupported");
+            TraceEventType eventType = parseEventType(range.lowerEndpoint().value(0));
+            if (range.upperEndpoint().size() == 1)
+            {
+                tracing().eraseEvents(txnId, eventType);
+            }
+            else
+            {
+                long before = range.upperEndpoint().value(1);
+                tracing().eraseEventsBefore(txnId, eventType, before);
+            }
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+
+        @Override
+        public DataSet data()
+        {
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing().forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    e.messages().forEach(m -> {
+                        dataSet.row(txnId.toString(), eventType.name(), e.atMicros, m.atNanos - e.atNanos)

Review Comment:
   last column was supposed to be micros, not nanos (`id_micros, at_micros`)
   is `atMicros` also on the right spot?



##########
src/java/org/apache/cassandra/service/accord/AccordCommandStore.java:
##########
@@ -19,6 +19,7 @@
 package org.apache.cassandra.service.accord;
 
 import java.util.List;
+import java.util.Map;

Review Comment:
   nit: unused imports 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2183797192


##########
src/java/org/apache/cassandra/service/accord/AccordTracing.java:
##########
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.AbstractList;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Predicate;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.Tracing;
+import accord.api.TraceEventType;
+import accord.local.CommandStore;
+import accord.primitives.TxnId;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.NoSpamLogger;
+
+public class AccordTracing
+{
+    private static final int MAX_EVENTS = 10000;
+    private static final Logger logger = LoggerFactory.getLogger(AccordTracing.class);
+    private static final NoSpamLogger noSpamLogger = NoSpamLogger.getLogger(logger, 1L, TimeUnit.MINUTES);
+
+    public interface ConsumeState
+    {
+        void accept(TxnId txnId, TraceEventType eventType, int permits, List<Event> events);
+    }
+
+    public static class Message
+    {
+        public final long atNanos;
+        public final int commandStoreId;
+        public final String message;
+
+        Message(int commandStoreId, String message, long atLeastNanos)
+        {
+            this.commandStoreId = commandStoreId;
+            this.message = message;
+            this.atNanos = Math.max(atLeastNanos, Clock.Global.nanoTime());
+        }
+
+        @Override
+        public String toString()
+        {
+            return message;
+        }
+    }
+
+    public static class Event implements Tracing, Comparable<Event>
+    {
+        public final long atMicros = uniqueNowMicros();
+        public final long atNanos = Clock.Global.nanoTime();
+        final List<Message> messages = new ArrayList<>();
+
+        @Override
+        public void trace(CommandStore commandStore, String s)
+        {
+            long prevNanos = messages.isEmpty() ? 0 : messages.get(messages.size() - 1).atNanos;
+            int id = commandStore == null ? -1 : commandStore.id();
+            if (s.length() > 1000)
+                s = s.substring(0, 1000);
+            messages.add(new Message(id, s, prevNanos + 1));
+        }
+
+        @Override
+        public int compareTo(Event that)
+        {
+            return Long.compareUnsigned(this.atMicros, that.atMicros);
+        }
+
+        public List<Message> messages()
+        {
+            return Collections.unmodifiableList(messages);
+        }
+    }
+
+    static class TraceState extends AbstractList<Event>
+    {
+        int permits;
+        int size;
+        Event[] events;
+
+        void addInternal(Event event)
+        {
+            if (events == null) events = new Event[10];
+            else if (size == events.length) events = new Event[size * 2];

Review Comment:
   Don't we need to copy old events over?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184541532


##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +
+                        "  command_store_id int,\n" +
+                        "  message text,\n" +
+                        "  PRIMARY KEY (txn_id, event_type, id_micros, at_micros)" +
+                        ')', UTF8Type.instance));
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().eraseEvents(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.lowerEndpoint().size() != 1) throw invalidRequest("Deletion restricted by lower bound on id_micros or at_micros is unsupported");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED && range.upperEndpoint().size() == 1)) throw invalidRequest("Range deletion must specify one event_type");
+            if (!range.upperEndpoint().value(0).equals(range.lowerEndpoint().value(0))) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.upperEndpoint().size() > 2) throw invalidRequest("Deletion restricted by upper bound on at_micros is unsupported");
+            TraceEventType eventType = parseEventType(range.lowerEndpoint().value(0));
+            if (range.upperEndpoint().size() == 1)
+            {
+                tracing().eraseEvents(txnId, eventType);
+            }
+            else
+            {
+                long before = range.upperEndpoint().value(1);
+                tracing().eraseEventsBefore(txnId, eventType, before);
+            }
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+
+        @Override
+        public DataSet data()
+        {
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing().forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    e.messages().forEach(m -> {
+                        dataSet.row(txnId.toString(), eventType.name(), e.atMicros, m.atNanos - e.atNanos)

Review Comment:
   yes, but I have renamed Event.atMicros to Event.idMicros for consistency/clarity



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184546606


##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +
+                        "  command_store_id int,\n" +
+                        "  message text,\n" +
+                        "  PRIMARY KEY (txn_id, event_type, id_micros, at_micros)" +
+                        ')', UTF8Type.instance));
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().eraseEvents(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.lowerEndpoint().size() != 1) throw invalidRequest("Deletion restricted by lower bound on id_micros or at_micros is unsupported");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED && range.upperEndpoint().size() == 1)) throw invalidRequest("Range deletion must specify one event_type");
+            if (!range.upperEndpoint().value(0).equals(range.lowerEndpoint().value(0))) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.upperEndpoint().size() > 2) throw invalidRequest("Deletion restricted by upper bound on at_micros is unsupported");
+            TraceEventType eventType = parseEventType(range.lowerEndpoint().value(0));
+            if (range.upperEndpoint().size() == 1)
+            {
+                tracing().eraseEvents(txnId, eventType);
+            }
+            else
+            {
+                long before = range.upperEndpoint().value(1);
+                tracing().eraseEventsBefore(txnId, eventType, before);
+            }
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+
+        @Override
+        public DataSet data()
+        {
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing().forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    e.messages().forEach(m -> {
+                        dataSet.row(txnId.toString(), eventType.name(), e.atMicros, m.atNanos - e.atNanos)
+                               .column("command_store_id", m.commandStoreId)
+                               .column("message", m.message);
+                    });
+                });
+            });
+            return dataSet;
+        }
+    }
+
+    public static final class TxnTable extends AbstractVirtualTable implements AbstractVirtualTable.DataSet
+    {
+        static class Entry
+        {
+            final int commandStoreId;
+            final Command command;
+
+            Entry(int commandStoreId, Command command)
+            {
+                this.commandStoreId = commandStoreId;
+                this.command = command;
+            }
+        }
+        private TxnTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN,
+                        "Accord per-CommandStore Transaction State",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  command_store_id int,\n" +
+                        "  save_status text,\n" +
+                        "  route text,\n" +
+                        "  durability text,\n" +
+                        "  execute_at text,\n" +
+                        "  txn text,\n" +
+                        "  deps text,\n" +
+                        "  waiting_on text,\n" +
+                        "  writes text,\n" +
+                        "  result text,\n" +
+                        "  participants_owns text,\n" +
+                        "  participants_touches text,\n" +
+                        "  participants_has_touched text,\n" +
+                        "  participants_executes text,\n" +
+                        "  participants_waits_on text,\n" +
+                        "  PRIMARY KEY (txn_id, command_store_id)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            return this;
+        }
+
+        @Override
+        public boolean isEmpty()
+        {
+            return false;
+        }
+
+        @Override
+        public Partition getPartition(DecoratedKey partitionKey)
+        {
+            String txnIdStr = UTF8Type.instance.compose(partitionKey.getKey());
+            TxnId txnId = TxnId.parse(txnIdStr);
+
+            List<Entry> commands = new ArrayList<>();
+            AccordService.instance().node().commandStores().forEachCommandStore(store -> {
+                Command command = ((AccordCommandStore)store).loadCommand(txnId);
+                if (command != null)
+                    commands.add(new Entry(store.id(), command));
+            });
+
+            if (commands.isEmpty())
+                return null;
+
+            SimpleDataSet ds = new SimpleDataSet(metadata);
+            for (Entry e : commands)
+            {
+                Command command = e.command;
+                ds.row(txnIdStr, e.commandStoreId)
+                  .column("save_status", Objects.toString(command.saveStatus()))
+                  .column("route", Objects.toString(command.route()))
+                  .column("participants_owns", toStr(command, StoreParticipants::owns, StoreParticipants::stillOwns))
+                  .column("participants_touches", toStr(command, StoreParticipants::touches, StoreParticipants::stillTouches))
+                  .column("participants_has_touched", Objects.toString(command.participants().hasTouched()))
+                  .column("participants_executes", toStr(command, StoreParticipants::executes, StoreParticipants::stillExecutes))
+                  .column("participants_waits_on", toStr(command, StoreParticipants::waitsOn, StoreParticipants::stillWaitsOn))
+                  .column("durability", Objects.toString(command.durability()))
+                  .column("execute_at", Objects.toString(command.executeAt()))

Review Comment:
   It's part of the `WaitingOn` collection here, which is perhaps confusing, but it looks like we don't print it as part of that so yes I will add an additional row.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184503097


##########
src/java/org/apache/cassandra/service/accord/AccordTracing.java:
##########
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.AbstractList;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Predicate;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.Tracing;
+import accord.api.TraceEventType;
+import accord.local.CommandStore;
+import accord.primitives.TxnId;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.NoSpamLogger;
+
+public class AccordTracing
+{
+    private static final int MAX_EVENTS = 10000;
+    private static final Logger logger = LoggerFactory.getLogger(AccordTracing.class);
+    private static final NoSpamLogger noSpamLogger = NoSpamLogger.getLogger(logger, 1L, TimeUnit.MINUTES);
+
+    public interface ConsumeState
+    {
+        void accept(TxnId txnId, TraceEventType eventType, int permits, List<Event> events);
+    }
+
+    public static class Message
+    {
+        public final long atNanos;
+        public final int commandStoreId;
+        public final String message;
+
+        Message(int commandStoreId, String message, long atLeastNanos)
+        {
+            this.commandStoreId = commandStoreId;
+            this.message = message;
+            this.atNanos = Math.max(atLeastNanos, Clock.Global.nanoTime());
+        }
+
+        @Override
+        public String toString()
+        {
+            return message;
+        }
+    }
+
+    public static class Event implements Tracing, Comparable<Event>
+    {
+        public final long atMicros = uniqueNowMicros();
+        public final long atNanos = Clock.Global.nanoTime();
+        final List<Message> messages = new ArrayList<>();
+
+        @Override
+        public void trace(CommandStore commandStore, String s)
+        {
+            long prevNanos = messages.isEmpty() ? 0 : messages.get(messages.size() - 1).atNanos;
+            int id = commandStore == null ? -1 : commandStore.id();
+            if (s.length() > 1000)
+                s = s.substring(0, 1000);
+            messages.add(new Message(id, s, prevNanos + 1));
+        }
+
+        @Override
+        public int compareTo(Event that)
+        {
+            return Long.compareUnsigned(this.atMicros, that.atMicros);
+        }
+
+        public List<Message> messages()
+        {
+            return Collections.unmodifiableList(messages);
+        }
+    }
+
+    static class TraceState extends AbstractList<Event>
+    {
+        int permits;
+        int size;
+        Event[] events;
+
+        void addInternal(Event event)
+        {
+            if (events == null) events = new Event[10];
+            else if (size == events.length) events = new Event[size * 2];

Review Comment:
   Heh, good catch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184538980


##########
src/java/org/apache/cassandra/service/accord/AccordCommandStore.java:
##########
@@ -19,6 +19,7 @@
 package org.apache.cassandra.service.accord;
 
 import java.util.List;
+import java.util.Map;

Review Comment:
   this has already been addressed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184544291


##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -577,6 +603,354 @@ public DataSet data()
         }
     }
 
+    public static final class TxnTraceTable extends AbstractMutableVirtualTable
+    {
+        private TxnTraceTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACE,
+                        "Accord Transaction Trace Configuration",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  event_type text,\n" +
+                        "  permits int,\n" +
+                        "  PRIMARY KEY (txn_id, event_type)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            AccordTracing tracing = ((AccordAgent)AccordService.instance().agent()).tracing();
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing.forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    dataSet.row(txnId.toString(), eventType).column("permits", permits);
+                });
+            });
+            return dataSet;
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().erasePermits(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED)) throw invalidRequest("Range deletion must specify one event_type");
+            if (range.lowerEndpoint().size() != 1 || range.upperEndpoint().size() != 1) throw invalidRequest("Invalid number of range deletion components");
+            tracing().erasePermits(txnId, parseEventType(range.lowerEndpoint().value(0)));
+        }
+
+        @Override
+        protected void applyColumnDeletion(ColumnValues partitionKey, ColumnValues clusteringColumns, String columnName)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            tracing().erasePermits(txnId, eventType);
+        }
+
+        @Override
+        protected void applyColumnUpdate(ColumnValues partitionKey, ColumnValues clusteringColumns, Optional<ColumnValue> columnValue)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            TraceEventType eventType = parseEventType(clusteringColumns.value(0));
+            if (columnValue.isEmpty()) tracing().erasePermits(txnId, eventType);
+            else tracing().setPermits(txnId, eventType, columnValue.get().value());
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+    }
+
+    public static final class TxnTracesTable extends AbstractMutableVirtualTable
+    {
+        private TxnTracesTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN_TRACES,
+                        "Accord Transaction Traces",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  id_micros bigint,\n" +
+                        "  event_type text,\n" +
+                        "  at_micros bigint,\n" +
+                        "  command_store_id int,\n" +
+                        "  message text,\n" +
+                        "  PRIMARY KEY (txn_id, event_type, id_micros, at_micros)" +
+                        ')', UTF8Type.instance));
+        }
+
+        private AccordTracing tracing()
+        {
+            return ((AccordAgent)AccordService.instance().agent()).tracing();
+        }
+
+        @Override
+        protected void applyPartitionDeletion(ColumnValues partitionKey)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            tracing().eraseEvents(txnId);
+        }
+
+        @Override
+        protected void applyRangeTombstone(ColumnValues partitionKey, Range<ColumnValues> range)
+        {
+            TxnId txnId = TxnId.parse(partitionKey.value(0));
+            if (!range.hasLowerBound() || range.lowerBoundType() != BoundType.CLOSED) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.lowerEndpoint().size() != 1) throw invalidRequest("Deletion restricted by lower bound on id_micros or at_micros is unsupported");
+            if (!range.hasUpperBound() || (range.upperBoundType() != BoundType.CLOSED && range.upperEndpoint().size() == 1)) throw invalidRequest("Range deletion must specify one event_type");
+            if (!range.upperEndpoint().value(0).equals(range.lowerEndpoint().value(0))) throw invalidRequest("May restrict deletion by at most one event_type");
+            if (range.upperEndpoint().size() > 2) throw invalidRequest("Deletion restricted by upper bound on at_micros is unsupported");
+            TraceEventType eventType = parseEventType(range.lowerEndpoint().value(0));
+            if (range.upperEndpoint().size() == 1)
+            {
+                tracing().eraseEvents(txnId, eventType);
+            }
+            else
+            {
+                long before = range.upperEndpoint().value(1);
+                tracing().eraseEventsBefore(txnId, eventType, before);
+            }
+        }
+
+        @Override
+        public void truncate()
+        {
+            tracing().eraseAllEvents();
+        }
+
+        @Override
+        public DataSet data()
+        {
+            SimpleDataSet dataSet = new SimpleDataSet(metadata());
+            tracing().forEach(id -> true, (txnId, eventType, permits, events) -> {
+                events.forEach(e -> {
+                    e.messages().forEach(m -> {
+                        dataSet.row(txnId.toString(), eventType.name(), e.atMicros, m.atNanos - e.atNanos)
+                               .column("command_store_id", m.commandStoreId)
+                               .column("message", m.message);
+                    });
+                });
+            });
+            return dataSet;
+        }
+    }
+
+    public static final class TxnTable extends AbstractVirtualTable implements AbstractVirtualTable.DataSet
+    {
+        static class Entry
+        {
+            final int commandStoreId;
+            final Command command;
+
+            Entry(int commandStoreId, Command command)
+            {
+                this.commandStoreId = commandStoreId;
+                this.command = command;
+            }
+        }
+        private TxnTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, TXN,
+                        "Accord per-CommandStore Transaction State",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  command_store_id int,\n" +
+                        "  save_status text,\n" +
+                        "  route text,\n" +
+                        "  durability text,\n" +
+                        "  execute_at text,\n" +
+                        "  txn text,\n" +
+                        "  deps text,\n" +
+                        "  waiting_on text,\n" +
+                        "  writes text,\n" +
+                        "  result text,\n" +
+                        "  participants_owns text,\n" +
+                        "  participants_touches text,\n" +
+                        "  participants_has_touched text,\n" +
+                        "  participants_executes text,\n" +
+                        "  participants_waits_on text,\n" +
+                        "  PRIMARY KEY (txn_id, command_store_id)" +
+                        ')', UTF8Type.instance));
+        }
+
+        @Override
+        public DataSet data()
+        {
+            return this;
+        }
+
+        @Override
+        public boolean isEmpty()
+        {
+            return false;
+        }
+
+        @Override
+        public Partition getPartition(DecoratedKey partitionKey)
+        {
+            String txnIdStr = UTF8Type.instance.compose(partitionKey.getKey());
+            TxnId txnId = TxnId.parse(txnIdStr);
+
+            List<Entry> commands = new ArrayList<>();
+            AccordService.instance().node().commandStores().forEachCommandStore(store -> {
+                Command command = ((AccordCommandStore)store).loadCommand(txnId);
+                if (command != null)
+                    commands.add(new Entry(store.id(), command));
+            });
+
+            if (commands.isEmpty())
+                return null;
+
+            SimpleDataSet ds = new SimpleDataSet(metadata);
+            for (Entry e : commands)
+            {
+                Command command = e.command;
+                ds.row(txnIdStr, e.commandStoreId)
+                  .column("save_status", Objects.toString(command.saveStatus()))
+                  .column("route", Objects.toString(command.route()))
+                  .column("participants_owns", toStr(command, StoreParticipants::owns, StoreParticipants::stillOwns))
+                  .column("participants_touches", toStr(command, StoreParticipants::touches, StoreParticipants::stillTouches))
+                  .column("participants_has_touched", Objects.toString(command.participants().hasTouched()))
+                  .column("participants_executes", toStr(command, StoreParticipants::executes, StoreParticipants::stillExecutes))
+                  .column("participants_waits_on", toStr(command, StoreParticipants::waitsOn, StoreParticipants::stillWaitsOn))
+                  .column("durability", Objects.toString(command.durability()))
+                  .column("execute_at", Objects.toString(command.executeAt()))
+                  .column("txn", Objects.toString(command.partialTxn()))
+                  .column("deps", Objects.toString(command.partialDeps()))
+                  .column("waiting_on", Objects.toString(command.waitingOn()))
+                  .column("writes", Objects.toString(command.writes()))
+                  .column("result", Objects.toString(command.result()));
+            }
+
+            return ds.getPartition(partitionKey);
+        }
+
+        @Override
+        public Iterator<Partition> getPartitions(DataRange range)
+        {
+            throw new UnsupportedOperationException();
+        }
+    }
+
+    public static final class JournalTable extends AbstractVirtualTable implements AbstractVirtualTable.DataSet
+    {
+        static class Entry
+        {
+            final int commandStoreId;
+            final long segment;
+            final int position;
+            final CommandChange.Builder builder;
+
+            Entry(int commandStoreId, long segment, int position, CommandChange.Builder builder)
+            {
+                this.commandStoreId = commandStoreId;
+                this.segment = segment;
+                this.position = position;
+                this.builder = builder;
+            }
+        }
+
+        private JournalTable()
+        {
+            super(parse(VIRTUAL_ACCORD_DEBUG, JOURNAL,
+                        "Accord per-CommandStore Journal State",
+                        "CREATE TABLE %s (\n" +
+                        "  txn_id text,\n" +
+                        "  command_store_id int,\n" +
+                        "  segment bigint,\n" +
+                        "  segment_position int,\n" +
+                        "  save_status text,\n" +
+                        "  route text,\n" +
+                        "  durability text,\n" +
+                        "  execute_at text,\n" +
+                        "  executes_at_least text,\n" +
+                        "  txn text,\n" +
+                        "  deps text,\n" +

Review Comment:
   We cannot easily report waiting_on from the journal because we don't necessarily have the deps that coincide without potentially reporting information from another segment. I agree it would be nice to report it at some point, but requires more thought than I have time for right now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith merged PR #4225:
URL: https://github.com/apache/cassandra/pull/4225


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2183790937


##########
test/unit/org/apache/cassandra/db/virtual/AccordDebugKeyspaceTest.java:
##########
@@ -85,6 +110,48 @@ public void unknownIsEmpty()
     {
         createTable("CREATE TABLE %s (k int, c int, v int, PRIMARY KEY (k, c)) WITH transactional_mode = 'full'");
         assertRows(execute(QUERY_TXN_BLOCKED_BY, TxnId.NONE.toString()));
+        assertRows(execute(QUERY_TXN, TxnId.NONE.toString()));
+        assertRows(execute(QUERY_JOURNAL, TxnId.NONE.toString()));
+    }
+
+    @Test
+    public void tracing()
+    {
+        // simple test to confirm basic tracing functionality works, doesn't validate specific behaviours only requesting/querying/erasing
+        AccordMsgFilter filter = new AccordMsgFilter();
+        MessagingService.instance().outboundSink.add(filter);
+        try
+        {
+            String tableName = createTable("CREATE TABLE %s (k int, c int, v int, PRIMARY KEY (k, c)) WITH transactional_mode = 'full'");
+            var accord = accord();
+            DatabaseDescriptor.getAccord().fetch_txn = "1s";
+            TxnId id = accord.node().nextTxnId(Txn.Kind.Write, Routable.Domain.Key);
+            Txn txn = createTxn(wrapInTxn(String.format("INSERT INTO %s.%s(k, c, v) VALUES (?, ?, ?)", KEYSPACE, tableName)), 0, 0, 0);
+
+            execute(SET_TRACE, 1, id.toString(), "PROGRESS");
+            accord.node().coordinate(id, txn);
+            filter.preAccept.awaitThrowUncheckedOnInterrupt();
+
+            filter.apply.awaitThrowUncheckedOnInterrupt();
+            spinUntilSuccess(() -> Assertions.assertThat(execute(QUERY_TRACES, id.toString(), "PROGRESS").size()).isGreaterThan(0));
+            execute(ERASE_TRACES1, id.toString(), "PROGRESS", Long.MAX_VALUE);
+            Assertions.assertThat(execute(QUERY_TRACES, id.toString(), "PROGRESS").size()).isEqualTo(0);
+            // just check other variants don't fail
+            execute(ERASE_TRACES2, id.toString(), "PROGRESS");

Review Comment:
   ```
   execute(ERASE_TRACES2, id.toString(), "FETCH");
   ```
   
   would throw an NPE 
   
   ```
   
   java.lang.NullPointerException
   	at org.apache.cassandra.service.accord.AccordTracing.lambda$eraseEvents$4(AccordTracing.java:264)
   	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932)
   	at org.apache.cassandra.service.accord.AccordTracing.eraseEvents(AccordTracing.java:260)
   	at org.apache.cassandra.db.virtual.AccordDebugKeyspace$TxnTracesTable.applyRangeTombstone(AccordDebugKeyspace.java:720)
   	at org.apache.cassandra.db.virtual.AbstractMutableVirtualTable.lambda$apply$2(AbstractMutableVirtualTable.java:101)
   	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
   	at org.apache.cassandra.db.virtual.AbstractMutableVirtualTable.apply(AbstractMutableVirtualTable.java:101)
   	at org.apache.cassandra.db.virtual.VirtualMutation.lambda$apply$0(VirtualMutation.java:64)
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184538451


##########
src/java/org/apache/cassandra/service/accord/AccordTracing.java:
##########
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.AbstractList;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Predicate;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.Tracing;
+import accord.api.TraceEventType;
+import accord.local.CommandStore;
+import accord.primitives.TxnId;
+import org.apache.cassandra.utils.Clock;
+import org.apache.cassandra.utils.NoSpamLogger;
+
+public class AccordTracing
+{
+    private static final int MAX_EVENTS = 10000;
+    private static final Logger logger = LoggerFactory.getLogger(AccordTracing.class);
+    private static final NoSpamLogger noSpamLogger = NoSpamLogger.getLogger(logger, 1L, TimeUnit.MINUTES);
+
+    public interface ConsumeState
+    {
+        void accept(TxnId txnId, TraceEventType eventType, int permits, List<Event> events);
+    }
+
+    public static class Message
+    {
+        public final long atNanos;
+        public final int commandStoreId;
+        public final String message;
+
+        Message(int commandStoreId, String message, long atLeastNanos)
+        {
+            this.commandStoreId = commandStoreId;
+            this.message = message;
+            this.atNanos = Math.max(atLeastNanos, Clock.Global.nanoTime());
+        }
+
+        @Override
+        public String toString()
+        {
+            return message;
+        }
+    }
+
+    public static class Event implements Tracing, Comparable<Event>
+    {
+        public final long atMicros = uniqueNowMicros();
+        public final long atNanos = Clock.Global.nanoTime();
+        final List<Message> messages = new ArrayList<>();
+
+        @Override
+        public void trace(CommandStore commandStore, String s)
+        {
+            long prevNanos = messages.isEmpty() ? 0 : messages.get(messages.size() - 1).atNanos;

Review Comment:
   atNanos is just the time we started collecting events. nanoTime can't go backwards, 0 logically makes clear we don't want to adjust the value that is taken, since the first event can reasonably coincide with the start of collecting events.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "ifesdjeen (via GitHub)" <gi...@apache.org>.
ifesdjeen commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2183787696


##########
test/unit/org/apache/cassandra/db/virtual/AccordDebugKeyspaceTest.java:
##########
@@ -65,6 +66,30 @@ public class AccordDebugKeyspaceTest extends CQLTester
     private static final String QUERY_TXN_BLOCKED_BY =
         String.format("SELECT * FROM %s.%s WHERE txn_id=?", SchemaConstants.VIRTUAL_ACCORD_DEBUG, AccordDebugKeyspace.TXN_BLOCKED_BY);
 
+    private static final String QUERY_TXN =
+        String.format("SELECT txn_id, save_status FROM %s.%s WHERE txn_id=?", SchemaConstants.VIRTUAL_ACCORD_DEBUG, AccordDebugKeyspace.TXN);
+
+    private static final String QUERY_JOURNAL =
+        String.format("SELECT txn_id, save_status FROM %s.%s WHERE txn_id=?", SchemaConstants.VIRTUAL_ACCORD_DEBUG, AccordDebugKeyspace.JOURNAL);
+
+    private static final String SET_TRACE =
+        String.format("UPDATE %s.%s SET permits = ? WHERE txn_id = ? AND event_type = ?", SchemaConstants.VIRTUAL_ACCORD_DEBUG, AccordDebugKeyspace.TXN_TRACE);

Review Comment:
   Should we add an example / test for 
   
   ```
   private static final String UNSET_TRACE =
       String.format("DELETE FROM %s.%s WHERE txn_id = ?", SchemaConstants.VIRTUAL_ACCORD_DEBUG, AccordDebugKeyspace.TXN_TRACE);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org


Re: [PR] Debug txn journal [cassandra]

Posted by "belliottsmith (via GitHub)" <gi...@apache.org>.
belliottsmith commented on code in PR #4225:
URL: https://github.com/apache/cassandra/pull/4225#discussion_r2184542824


##########
src/java/org/apache/cassandra/db/virtual/AccordDebugKeyspace.java:
##########
@@ -95,6 +112,10 @@ public class AccordDebugKeyspace extends VirtualKeyspace
     public static final String REDUNDANT_BEFORE   = "redundant_before";
     public static final String REJECT_BEFORE      = "reject_before";
     public static final String TXN_BLOCKED_BY     = "txn_blocked_by";
+    public static final String TXN                = "txn";
+    public static final String JOURNAL            = "journal";
+    public static final String TXN_TRACE          = "txn_trace";
+    public static final String TXN_TRACES         = "txn_traces";

Review Comment:
   I prefer shorter names for less typing when debugging, and traces is what you produce when tracing. But it is a weak preference, so happy to be overruled. We can revisit all of this later as well, as this is not a stable API



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org