You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "ibessonov (via GitHub)" <gi...@apache.org> on 2023/06/30 08:58:03 UTC
[GitHub] [ignite-3] ibessonov opened a new pull request, #2273: IGNITE-15568 Implement batch log updates for RAFT.
ibessonov opened a new pull request, #2273:
URL: https://github.com/apache/ignite-3/pull/2273
https://issues.apache.org/jira/browse/IGNITE-15568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov merged pull request #2273: IGNITE-19946 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov merged PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] alievmirza commented on a diff in pull request #2273: IGNITE-15568 Implement batch log updates for RAFT.
Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1258149634
##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/disruptor/StripedDisruptor.java:
##########
@@ -214,7 +223,7 @@ void unsubscribe(NodeId nodeId) {
assert handler != null : format("Group of the event is unsupported [nodeId={}, event={}]", event.nodeId(), event);
//TODO: IGNITE-15568 endOfBatch should be set to true to prevent caching tasks until IGNITE-15568 has fixed.
- handler.onEvent(event, sequence, subscribers.size() > 1 ? true : endOfBatch);
+ handler.onEvent(event, sequence, endOfBatch || subscribers.size() >= 1 && !supportsBatches);
Review Comment:
How is this possible to `subscribers.size() >= 1` be false, taking into account assertion above?
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/DefaultLogStorageFactory.java:
##########
@@ -69,16 +70,24 @@ public class DefaultLogStorageFactory implements LogStorageFactory {
/** Data column family handle. */
private ColumnFamilyHandle dataHandle;
+ /**
+ * Thread-local batch instance, used by {@link RocksDbSharedLogStorage#appendEntriesToBatch(List)} and
+ * {@link RocksDbSharedLogStorage#commitWriteBatch()}.
+ * <br>
+ * Shared between instances to provide more efficient way of executing batch updates.
+ */
+ @SuppressWarnings("ThreadLocalNotStaticFinal")
+ private final ThreadLocal<WriteBatch> threadLocalWriteBatch = new ThreadLocal<>();
+
/**
* Constructor.
*
* @param path Path to the storage.
*/
- public DefaultLogStorageFactory(Path path) {
+ public DefaultLogStorageFactory(Path path, int stripes) {
Review Comment:
`stripes` param is not used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2273: IGNITE-15568 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1255639570
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/StripeAwareLogManager.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.raft.storage.impl;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.raft.jraft.Status;
+import org.apache.ignite.raft.jraft.entity.LogEntry;
+import org.apache.ignite.raft.jraft.entity.LogId;
+import org.apache.ignite.raft.jraft.error.RaftError;
+import org.apache.ignite.raft.jraft.option.LogManagerOptions;
+import org.apache.ignite.raft.jraft.storage.LogStorage;
+import org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl;
+
+/**
+ * Log manager that enables batch processing of log entries from different partitions within a stripe.
+ * <br>
+ * Upon each flush of the log, it will also trigger flush on all the other log storages, that belong to the same stripe in
+ * corresponding {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+public class StripeAwareLogManager extends LogManagerImpl {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(LogManagerImpl.class);
+
+ /** Log storage instance. */
+ private LogStorage logStorage;
+
+ /** Stripe, that corresponds to the current log storage instance. */
+ private final Stripe stripe;
+
+ /** Size threshold of log entries list, that will trigger the flush upon the excess. */
+ private int maxAppendBufferSize;
+
+ /**
+ * Whether the log storage is a {@link RocksDbSharedLogStorage} or not.
+ * It requires special treatment in order to better optimize writes.
+ */
+ private boolean sharedLogStorage;
+
+ /**
+ * Constructor.
+ *
+ * @param stripe Stripe that corresponds to a worker thread in {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+ public StripeAwareLogManager(Stripe stripe) {
+ this.stripe = stripe;
+ }
+
+ @Override
+ public boolean init(LogManagerOptions opts) {
+ LogStorage logStorage = opts.getLogStorage();
+
+ this.sharedLogStorage = logStorage instanceof RocksDbSharedLogStorage;
Review Comment:
Yes, it should, there are benefits for them as well. That's actually how it's supposed to be according to jraft, they don't have shared logs, they prepare batches for a single log storage only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2273: IGNITE-19946 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1259299932
##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/disruptor/StripedDisruptor.java:
##########
@@ -214,7 +223,7 @@ void unsubscribe(NodeId nodeId) {
assert handler != null : format("Group of the event is unsupported [nodeId={}, event={}]", event.nodeId(), event);
//TODO: IGNITE-15568 endOfBatch should be set to true to prevent caching tasks until IGNITE-15568 has fixed.
- handler.onEvent(event, sequence, subscribers.size() > 1 ? true : endOfBatch);
+ handler.onEvent(event, sequence, endOfBatch || subscribers.size() >= 1 && !supportsBatches);
Review Comment:
Must be `> 1` I guess
##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/disruptor/StripedDisruptor.java:
##########
@@ -214,7 +223,7 @@ void unsubscribe(NodeId nodeId) {
assert handler != null : format("Group of the event is unsupported [nodeId={}, event={}]", event.nodeId(), event);
//TODO: IGNITE-15568 endOfBatch should be set to true to prevent caching tasks until IGNITE-15568 has fixed.
- handler.onEvent(event, sequence, subscribers.size() > 1 ? true : endOfBatch);
+ handler.onEvent(event, sequence, endOfBatch || subscribers.size() >= 1 && !supportsBatches);
Review Comment:
Must be `== 1` I guess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #2273: IGNITE-15568 Implement batch log updates for RAFT.
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1255612729
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/DefaultLogStorageFactory.java:
##########
@@ -69,16 +70,24 @@ public class DefaultLogStorageFactory implements LogStorageFactory {
/** Data column family handle. */
private ColumnFamilyHandle dataHandle;
+ /**
+ * Thread-local batch instance, used by {@link RocksDbSharedLogStorage#appendEntriesToBatch(List)} and
+ * {@link RocksDbSharedLogStorage#commitWriteBatch()}.
+ * <br>
+ * Shared between instances to provide more efficient way of executing batch updates.
+ */
+ @SuppressWarnings("ThreadLocalNotStaticFinal")
+ private final ThreadLocal<WriteBatch> threadLocalWriteBatch = new ThreadLocal<>();
Review Comment:
How many of them will be per-node giben that they are not static?
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/StripeAwareLogManager.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.raft.storage.impl;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.raft.jraft.Status;
+import org.apache.ignite.raft.jraft.entity.LogEntry;
+import org.apache.ignite.raft.jraft.entity.LogId;
+import org.apache.ignite.raft.jraft.error.RaftError;
+import org.apache.ignite.raft.jraft.option.LogManagerOptions;
+import org.apache.ignite.raft.jraft.storage.LogStorage;
+import org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl;
+
+/**
+ * Log manager that enables batch processing of log entries from different partitions within a stripe.
+ * <br>
+ * Upon each flush of the log, it will also trigger flush on all the other log storages, that belong to the same stripe in
+ * corresponding {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+public class StripeAwareLogManager extends LogManagerImpl {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(LogManagerImpl.class);
Review Comment:
Shouldn't it be StripeAwareLogManager.class?
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/StripeAwareLogManager.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.raft.storage.impl;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.raft.jraft.Status;
+import org.apache.ignite.raft.jraft.entity.LogEntry;
+import org.apache.ignite.raft.jraft.entity.LogId;
+import org.apache.ignite.raft.jraft.error.RaftError;
+import org.apache.ignite.raft.jraft.option.LogManagerOptions;
+import org.apache.ignite.raft.jraft.storage.LogStorage;
+import org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl;
+
+/**
+ * Log manager that enables batch processing of log entries from different partitions within a stripe.
+ * <br>
+ * Upon each flush of the log, it will also trigger flush on all the other log storages, that belong to the same stripe in
+ * corresponding {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+public class StripeAwareLogManager extends LogManagerImpl {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(LogManagerImpl.class);
+
+ /** Log storage instance. */
+ private LogStorage logStorage;
+
+ /** Stripe, that corresponds to the current log storage instance. */
+ private final Stripe stripe;
+
+ /** Size threshold of log entries list, that will trigger the flush upon the excess. */
+ private int maxAppendBufferSize;
+
+ /**
+ * Whether the log storage is a {@link RocksDbSharedLogStorage} or not.
+ * It requires special treatment in order to better optimize writes.
+ */
+ private boolean sharedLogStorage;
+
+ /**
+ * Constructor.
+ *
+ * @param stripe Stripe that corresponds to a worker thread in {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+ public StripeAwareLogManager(Stripe stripe) {
+ this.stripe = stripe;
+ }
+
+ @Override
+ public boolean init(LogManagerOptions opts) {
+ LogStorage logStorage = opts.getLogStorage();
+
+ this.sharedLogStorage = logStorage instanceof RocksDbSharedLogStorage;
Review Comment:
Should it work with non-shared storage?
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/StripeAwareLogManager.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.raft.storage.impl;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.raft.jraft.Status;
+import org.apache.ignite.raft.jraft.entity.LogEntry;
+import org.apache.ignite.raft.jraft.entity.LogId;
+import org.apache.ignite.raft.jraft.error.RaftError;
+import org.apache.ignite.raft.jraft.option.LogManagerOptions;
+import org.apache.ignite.raft.jraft.storage.LogStorage;
+import org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl;
+
+/**
+ * Log manager that enables batch processing of log entries from different partitions within a stripe.
+ * <br>
+ * Upon each flush of the log, it will also trigger flush on all the other log storages, that belong to the same stripe in
+ * corresponding {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+public class StripeAwareLogManager extends LogManagerImpl {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(LogManagerImpl.class);
+
+ /** Log storage instance. */
+ private LogStorage logStorage;
+
+ /** Stripe, that corresponds to the current log storage instance. */
+ private final Stripe stripe;
+
+ /** Size threshold of log entries list, that will trigger the flush upon the excess. */
+ private int maxAppendBufferSize;
+
+ /**
+ * Whether the log storage is a {@link RocksDbSharedLogStorage} or not.
+ * It requires special treatment in order to better optimize writes.
+ */
+ private boolean sharedLogStorage;
+
+ /**
+ * Constructor.
+ *
+ * @param stripe Stripe that corresponds to a worker thread in {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+ public StripeAwareLogManager(Stripe stripe) {
+ this.stripe = stripe;
+ }
+
+ @Override
+ public boolean init(LogManagerOptions opts) {
+ LogStorage logStorage = opts.getLogStorage();
+
+ this.sharedLogStorage = logStorage instanceof RocksDbSharedLogStorage;
+ this.logStorage = logStorage;
+ this.maxAppendBufferSize = opts.getRaftOptions().getMaxAppendBufferSize();
+
+ return super.init(opts);
+ }
+
+ /**
+ * Regular append to shared storage has been replaced with appending data into a batch. Data will later be "committed" by calling
+ * {@link StripeAwareAppendBatcher#commitWriteBatch()} on any of the log instances.
+ * The reason why is given in {@link Stripe}'s comments.
+ */
+ @Override
+ protected int appendToLogStorage(List<LogEntry> toAppend) {
+ if (sharedLogStorage) {
+ return ((RocksDbSharedLogStorage) logStorage).appendEntriesToBatch(toAppend) ? toAppend.size() : 0;
+ } else {
+ return logStorage.appendEntries(toAppend);
+ }
+ }
+
+ @Override
+ protected AppendBatcher newAppendBatcher(List<StableClosure> storages, int cap, LogId diskId) {
+ return new StripeAwareAppendBatcher(storages, cap, diskId);
+ }
+
+ /**
+ * Append batcher implementation that triggers flush on all log storages, that belong to the same stripe.
+ */
+ private class StripeAwareAppendBatcher extends AppendBatcher {
+ StripeAwareAppendBatcher(List<StableClosure> storage, int cap, LogId lastId) {
+ super(storage, cap, new ArrayList<>(), lastId);
+ }
+
+ private LogId lastIdCandidate;
+
+ /**
+ * Flush is simply delegated to the {@link Stripe}.
+ */
+ @Override
+ protected LogId flush() {
+ stripe.flush();
+
+ // Last ID is already updated at this point.
+ return lastId;
+ }
+
+ /**
+ * Delegates to {@link LogManagerImpl#appendToStorage(List)}.
+ */
+ void appendToStorage() {
+ assert size > 0;
+
+ lastIdCandidate = StripeAwareLogManager.super.appendToStorage(toAppend);
+ }
+
+ /**
+ * Delegates to {@link RocksDbSharedLogStorage#commitWriteBatch()} if it can. No-op otherwise.
+ */
+ void commitWriteBatch() {
+ if (sharedLogStorage) {
+ ((RocksDbSharedLogStorage) logStorage).commitWriteBatch();
+ }
+ }
+
+ /**
+ * Delegates to {@link LogManagerImpl#reportError(int, String, Object...)}.
+ */
+ void reportError(int code, String fmt, Object... args) {
+ StripeAwareLogManager.super.reportError(code, fmt, args);
+ }
+
+ /**
+ * Notifies storage stable closures and clears the batch. Based on the code from {@link AppendBatcher#flush()}.
+ */
+ void notifyClosures() {
+ lastId = lastIdCandidate;
+
+ for (int i = 0; i < this.size; i++) {
+ this.storage.get(i).getEntries().clear();
+ Status st = null;
+ try {
+ if (StripeAwareLogManager.super.hasError) {
+ st = new Status(RaftError.EIO, "Corrupted LogStorage");
+ } else {
+ st = Status.OK();
+ }
+ this.storage.get(i).run(st);
+ } catch (Throwable t) {
+ LOG.error("Fail to run closure with status: {}.", t, st);
+ }
+ }
+
+ toAppend.clear();
+ storage.clear();
+ size = 0;
+
+ setDiskId(lastId);
+ }
+
+ @Override
+ protected void append(StableClosure done) {
+ if (stripe.size >= cap || stripe.bufferSize >= maxAppendBufferSize) {
+ flush();
+ }
+
+ // This part is a little bit hacky. We must provide a valid buffer size delta to a stripe. The easiest way to do so it to treat
Review Comment:
I don't get it :\
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2273: IGNITE-19946 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1259299932
##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/disruptor/StripedDisruptor.java:
##########
@@ -214,7 +223,7 @@ void unsubscribe(NodeId nodeId) {
assert handler != null : format("Group of the event is unsupported [nodeId={}, event={}]", event.nodeId(), event);
//TODO: IGNITE-15568 endOfBatch should be set to true to prevent caching tasks until IGNITE-15568 has fixed.
- handler.onEvent(event, sequence, subscribers.size() > 1 ? true : endOfBatch);
+ handler.onEvent(event, sequence, endOfBatch || subscribers.size() >= 1 && !supportsBatches);
Review Comment:
I'll fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2273: IGNITE-15568 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1255636041
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/StripeAwareLogManager.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.raft.storage.impl;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.raft.jraft.Status;
+import org.apache.ignite.raft.jraft.entity.LogEntry;
+import org.apache.ignite.raft.jraft.entity.LogId;
+import org.apache.ignite.raft.jraft.error.RaftError;
+import org.apache.ignite.raft.jraft.option.LogManagerOptions;
+import org.apache.ignite.raft.jraft.storage.LogStorage;
+import org.apache.ignite.raft.jraft.storage.impl.LogManagerImpl;
+
+/**
+ * Log manager that enables batch processing of log entries from different partitions within a stripe.
+ * <br>
+ * Upon each flush of the log, it will also trigger flush on all the other log storages, that belong to the same stripe in
+ * corresponding {@link LogManagerOptions#getLogManagerDisruptor()}.
+ */
+public class StripeAwareLogManager extends LogManagerImpl {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(LogManagerImpl.class);
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2273: IGNITE-19946 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1259299361
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/DefaultLogStorageFactory.java:
##########
@@ -69,16 +70,24 @@ public class DefaultLogStorageFactory implements LogStorageFactory {
/** Data column family handle. */
private ColumnFamilyHandle dataHandle;
+ /**
+ * Thread-local batch instance, used by {@link RocksDbSharedLogStorage#appendEntriesToBatch(List)} and
+ * {@link RocksDbSharedLogStorage#commitWriteBatch()}.
+ * <br>
+ * Shared between instances to provide more efficient way of executing batch updates.
+ */
+ @SuppressWarnings("ThreadLocalNotStaticFinal")
+ private final ThreadLocal<WriteBatch> threadLocalWriteBatch = new ThreadLocal<>();
+
/**
* Constructor.
*
* @param path Path to the storage.
*/
- public DefaultLogStorageFactory(Path path) {
+ public DefaultLogStorageFactory(Path path, int stripes) {
Review Comment:
That's right, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #2273: IGNITE-15568 Implement batch log updates for RAFT.
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #2273:
URL: https://github.com/apache/ignite-3/pull/2273#discussion_r1255638474
##########
modules/raft/src/main/java/org/apache/ignite/internal/raft/storage/impl/DefaultLogStorageFactory.java:
##########
@@ -69,16 +70,24 @@ public class DefaultLogStorageFactory implements LogStorageFactory {
/** Data column family handle. */
private ColumnFamilyHandle dataHandle;
+ /**
+ * Thread-local batch instance, used by {@link RocksDbSharedLogStorage#appendEntriesToBatch(List)} and
+ * {@link RocksDbSharedLogStorage#commitWriteBatch()}.
+ * <br>
+ * Shared between instances to provide more efficient way of executing batch updates.
+ */
+ @SuppressWarnings("ThreadLocalNotStaticFinal")
+ private final ThreadLocal<WriteBatch> threadLocalWriteBatch = new ThreadLocal<>();
Review Comment:
Well, one per node. I can make it static, it will work, I just thought that non-static field makes more sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org