You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2021/06/25 06:45:12 UTC
[GitHub] [ignite] tkalkirill opened a new pull request #9198: IGNITE-14952
tkalkirill opened a new pull request #9198:
URL: https://github.com/apache/ignite/pull/9198
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on a change in pull request #9198: IGNITE-14952 Automatic release of WAL segments when reaching the DataStorageConfiguration#maxWalArchiveSize
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on a change in pull request #9198:
URL: https://github.com/apache/ignite/pull/9198#discussion_r664527318
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/aware/SegmentArchiveSizeStorage.java
##########
@@ -17,65 +17,132 @@
package org.apache.ignite.internal.processors.cache.persistence.wal.aware;
+import java.util.Map;
+import java.util.TreeMap;
+import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
/**
* Storage WAL archive size.
* Allows to track the exceeding of the maximum archive size.
*/
class SegmentArchiveSizeStorage {
- /** Current WAL archive size in bytes. */
+ /** Current WAL archive size in bytes. Guarded by {@code this}. */
private long curr;
- /** Reserved WAL archive size in bytes. */
- private long reserved;
+ /** Flag of interrupt waiting on this object. Guarded by {@code this}. */
+ private boolean interrupted;
+
+ /** Minimum size of the WAL archive in bytes. */
+ private final long minWalArchiveSize;
+
+ /** Maximum size of the WAL archive in bytes. */
+ private final long maxWalArchiveSize;
+
+ /**
+ * Segment sizes. Mapping: segment idx -> size in bytes. Guarded by {@code this}.
+ * {@code null} if {@link #maxWalArchiveSize} == {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ */
+ @Nullable private final Map<Long, Long> segmentSize;
- /** Flag of interrupt waiting on this object. */
- private volatile boolean interrupted;
+ /**
+ * Segment reservations storage.
+ * {@code null} if {@link #maxWalArchiveSize} == {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ */
+ @Nullable private final SegmentReservationStorage reservationStorage;
/**
- * Adding current WAL archive size in bytes.
+ * Constructor.
*
- * @param size Size in bytes.
+ * @param minWalArchiveSize Minimum size of the WAL archive in bytes.
+ * @param maxWalArchiveSize Maximum size of the WAL archive in bytes
+ * or {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ * @param reservationStorage Segment reservations storage.
*/
- synchronized void addCurrentSize(long size) {
- curr += size;
+ public SegmentArchiveSizeStorage(
+ long minWalArchiveSize,
+ long maxWalArchiveSize,
+ SegmentReservationStorage reservationStorage
+ ) {
+ this.minWalArchiveSize = minWalArchiveSize;
+ this.maxWalArchiveSize = maxWalArchiveSize;
- if (size > 0)
- notifyAll();
+ if (maxWalArchiveSize != UNLIMITED_WAL_ARCHIVE) {
+ segmentSize = new TreeMap<>();
+ this.reservationStorage = reservationStorage;
+ }
+ else {
+ segmentSize = null;
+ this.reservationStorage = null;
+ }
}
/**
- * Adding reserved WAL archive size in bytes.
- * Defines a hint to determine if the maximum size is exceeded before a new segment is archived.
+ * Adding the WAL segment size in the archive.
*
- * @param size Size in bytes.
+ * @param idx Absolut segment index.
+ * @param curr Current WAL archive size in bytes.
Review comment:
This javadoc is wrong and name of parameter is bad. Semantics of the parameter is about change of WAL archive size. Lets call it that way: `sizeChange`.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/aware/SegmentArchiveSizeStorage.java
##########
@@ -17,65 +17,132 @@
package org.apache.ignite.internal.processors.cache.persistence.wal.aware;
+import java.util.Map;
+import java.util.TreeMap;
+import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
/**
* Storage WAL archive size.
* Allows to track the exceeding of the maximum archive size.
*/
class SegmentArchiveSizeStorage {
- /** Current WAL archive size in bytes. */
+ /** Current WAL archive size in bytes. Guarded by {@code this}. */
private long curr;
Review comment:
Name of the field doesn't relate to its semantics. I suggest to rename it to something like `archiveSize` or `walArchiveSize`.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/aware/SegmentAware.java
##########
@@ -318,22 +329,13 @@ public boolean minLockIndex(long absIdx) {
}
/**
- * Adding current WAL archive size in bytes.
- *
- * @param size Size in bytes.
- */
- public void addCurrentWalArchiveSize(long size) {
- archiveSizeStorage.addCurrentSize(size);
- }
-
- /**
- * Adding reserved WAL archive size in bytes.
- * Defines a hint to determine if the maximum size is exceeded before a new segment is archived.
+ * Adding the WAL segment size in the archive.
*
- * @param size Size in bytes.
+ * @param idx Absolut segment index.
+ * @param curr Current WAL archive size in bytes.
*/
- public void addReservedWalArchiveSize(long size) {
- archiveSizeStorage.addReservedSize(size);
+ public void addSize(long idx, long curr) {
Review comment:
The same about semantics of the second parameter is here.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/aware/SegmentArchiveSizeStorage.java
##########
@@ -17,65 +17,132 @@
package org.apache.ignite.internal.processors.cache.persistence.wal.aware;
+import java.util.Map;
+import java.util.TreeMap;
+import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
/**
* Storage WAL archive size.
* Allows to track the exceeding of the maximum archive size.
*/
class SegmentArchiveSizeStorage {
- /** Current WAL archive size in bytes. */
+ /** Current WAL archive size in bytes. Guarded by {@code this}. */
private long curr;
- /** Reserved WAL archive size in bytes. */
- private long reserved;
+ /** Flag of interrupt waiting on this object. Guarded by {@code this}. */
+ private boolean interrupted;
+
+ /** Minimum size of the WAL archive in bytes. */
+ private final long minWalArchiveSize;
+
+ /** Maximum size of the WAL archive in bytes. */
+ private final long maxWalArchiveSize;
+
+ /**
+ * Segment sizes. Mapping: segment idx -> size in bytes. Guarded by {@code this}.
+ * {@code null} if {@link #maxWalArchiveSize} == {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ */
+ @Nullable private final Map<Long, Long> segmentSize;
- /** Flag of interrupt waiting on this object. */
- private volatile boolean interrupted;
+ /**
+ * Segment reservations storage.
+ * {@code null} if {@link #maxWalArchiveSize} == {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ */
+ @Nullable private final SegmentReservationStorage reservationStorage;
/**
- * Adding current WAL archive size in bytes.
+ * Constructor.
*
- * @param size Size in bytes.
+ * @param minWalArchiveSize Minimum size of the WAL archive in bytes.
+ * @param maxWalArchiveSize Maximum size of the WAL archive in bytes
+ * or {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ * @param reservationStorage Segment reservations storage.
*/
- synchronized void addCurrentSize(long size) {
- curr += size;
+ public SegmentArchiveSizeStorage(
+ long minWalArchiveSize,
+ long maxWalArchiveSize,
+ SegmentReservationStorage reservationStorage
+ ) {
+ this.minWalArchiveSize = minWalArchiveSize;
+ this.maxWalArchiveSize = maxWalArchiveSize;
- if (size > 0)
- notifyAll();
+ if (maxWalArchiveSize != UNLIMITED_WAL_ARCHIVE) {
+ segmentSize = new TreeMap<>();
+ this.reservationStorage = reservationStorage;
+ }
+ else {
+ segmentSize = null;
+ this.reservationStorage = null;
+ }
}
/**
- * Adding reserved WAL archive size in bytes.
- * Defines a hint to determine if the maximum size is exceeded before a new segment is archived.
+ * Adding the WAL segment size in the archive.
*
- * @param size Size in bytes.
+ * @param idx Absolut segment index.
+ * @param curr Current WAL archive size in bytes.
*/
- synchronized void addReservedSize(long size) {
- reserved += size;
+ void addSize(long idx, long curr) {
+ long releaseIdx = -1;
+
+ synchronized (this) {
+ this.curr += curr;
+
+ if (segmentSize != null) {
+ segmentSize.compute(idx, (i, size) -> {
+ long res = (size == null ? 0 : size) + curr;
+
+ return res == 0 ? null : res;
+ });
+ }
+
+ if (curr > 0) {
+ if (segmentSize != null && this.curr >= maxWalArchiveSize) {
+ long size = 0;
+
+ for (Map.Entry<Long, Long> e : segmentSize.entrySet()) {
+ releaseIdx = e.getKey();
+
+ if (this.curr - (size += e.getValue()) < minWalArchiveSize)
+ break;
+ }
+ }
- if (size > 0)
- notifyAll();
+ notifyAll();
+ }
+ }
+
+ if (releaseIdx != -1) {
+ assert reservationStorage != null;
Review comment:
Assert doesn't provide information on which `releaseIdx` it has failed. Please add a message to the assert clause.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/aware/SegmentArchiveSizeStorage.java
##########
@@ -17,65 +17,132 @@
package org.apache.ignite.internal.processors.cache.persistence.wal.aware;
+import java.util.Map;
+import java.util.TreeMap;
+import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.internal.IgniteInterruptedCheckedException;
+import org.jetbrains.annotations.Nullable;
+
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
/**
* Storage WAL archive size.
* Allows to track the exceeding of the maximum archive size.
*/
class SegmentArchiveSizeStorage {
- /** Current WAL archive size in bytes. */
+ /** Current WAL archive size in bytes. Guarded by {@code this}. */
private long curr;
- /** Reserved WAL archive size in bytes. */
- private long reserved;
+ /** Flag of interrupt waiting on this object. Guarded by {@code this}. */
+ private boolean interrupted;
+
+ /** Minimum size of the WAL archive in bytes. */
+ private final long minWalArchiveSize;
+
+ /** Maximum size of the WAL archive in bytes. */
+ private final long maxWalArchiveSize;
+
+ /**
+ * Segment sizes. Mapping: segment idx -> size in bytes. Guarded by {@code this}.
+ * {@code null} if {@link #maxWalArchiveSize} == {@link DataStorageConfiguration#UNLIMITED_WAL_ARCHIVE}.
+ */
+ @Nullable private final Map<Long, Long> segmentSize;
Review comment:
Field name is a bit misleading: it is a size of a segment, but sizes of all segments in archive dir.
How about renaming it to `segmentsSizes`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org