You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/11/30 12:27:06 UTC
[GitHub] [ignite] alex-plekhanov opened a new pull request #8513: IGNITE-13761 Segmented-LRU page replacement
alex-plekhanov opened a new pull request #8513:
URL: https://github.com/apache/ignite/pull/8513
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555540193
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/SegmentedLruPageList.java
##########
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Pages Segmented-LRU (SLRU) list implementation.
+ *
+ * @see PageReplacementMode#SEGMENTED_LRU
+ */
+public class SegmentedLruPageList {
+ /** Ratio to limit count of protected pages. */
+ private static final double PROTECTED_TO_TOTAL_PAGES_RATIO = 0.5;
+
+ /** Null page index. */
+ static final int NULL_IDX = -1;
+
+ /** Index of the head page of LRU list. */
+ private int headIdx = NULL_IDX;
+
+ /** Index of the tail page of LRU list. */
+ private int tailIdx = NULL_IDX;
+
+ /** Index of the tail page of probationary segment. */
+ private int probTailIdx = NULL_IDX;
+
+ /** Count of protected pages in the list. */
+ private int protectedPagesCnt;
+
+ /** Protected pages segment limit. */
+ private final int protectedPagesLimit;
+
+ /** Pointer to memory region to store links. */
+ private final long linksPtr;
+
+ /** Pointer to memory region to store protected flags. */
+ private final long flagsPtr;
+
+ /**
+ * @param totalPagesCnt Total pages count.
+ * @param memPtr Pointer to memory region.
+ */
+ public SegmentedLruPageList(int totalPagesCnt, long memPtr) {
+ linksPtr = memPtr;
+ flagsPtr = memPtr + (((long)totalPagesCnt) << 3);
+
+ GridUnsafe.setMemory(linksPtr, ((long)totalPagesCnt) << 3, (byte)0xFF);
+ GridUnsafe.setMemory(flagsPtr, (totalPagesCnt + 7) >> 3, (byte)0);
+
+ protectedPagesLimit = (int)(totalPagesCnt * PROTECTED_TO_TOTAL_PAGES_RATIO);
+ }
+
+ /**
+ * Remove page from the head of LRU list.
+ *
+ * @return Page index or {@code -1} if list is empty.
+ */
+ public synchronized int poll() {
Review comment:
this call is under wlock, isn`t it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] asfgit closed pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #8513:
URL: https://github.com/apache/ignite/pull/8513
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r556349467
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/ClockPageReplacementFlags.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import java.util.function.LongUnaryOperator;
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Clock page replacement algorithm implementation.
+ *
+ * @see PageReplacementMode#CLOCK
+ */
+public class ClockPageReplacementFlags {
+ /** Total pages count. */
+ private final int pagesCnt;
+
+ /** Index of the next candidate ("hand"). */
+ private volatile int curIdx;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r556273086
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/PageReplacementMode.java
##########
@@ -0,0 +1,85 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.ignite.configuration;
+
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Defines memory page replacement algorithm. A mode is set for a specific {@link DataRegionConfiguration}.
+ */
+public enum PageReplacementMode {
+ /**
+ * Random-LRU algorithm.
+ *
+ * Every time page is accessed, its timestamp gets updated. When a page fault occurs and it's required to replace
+ * some pages, the algorithm randomly chooses 5 pages from the page memory and evicts a page with the latest
+ * timestamp.
+ *
+ * This algorithm have zero maintenance cost, but not very effective in terms of finding next page to replace.
+ * Recommended to use in environments, where there are no page replacements (large enough data region to store all
Review comment:
yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555027799
##########
File path: modules/yardstick/src/main/java/org/apache/ignite/yardstick/cache/IgniteGetWithPageReplacementBenchmark.java
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.yardstick.cache;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Benchmark get operation with page replacement.
+ */
+public class IgniteGetWithPageReplacementBenchmark extends IgniteAbstractPageReplacementBenchmark {
Review comment:
why this test call page replacement?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555059167
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/PageReplacementMode.java
##########
@@ -0,0 +1,85 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.ignite.configuration;
+
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Defines memory page replacement algorithm. A mode is set for a specific {@link DataRegionConfiguration}.
+ */
+public enum PageReplacementMode {
+ /**
+ * Random-LRU algorithm.
+ *
+ * Every time page is accessed, its timestamp gets updated. When a page fault occurs and it's required to replace
+ * some pages, the algorithm randomly chooses 5 pages from the page memory and evicts a page with the latest
+ * timestamp.
+ *
+ * This algorithm have zero maintenance cost, but not very effective in terms of finding next page to replace.
+ * Recommended to use in environments, where there are no page replacements (large enough data region to store all
Review comment:
Which metric do you mean? PagesReplaceRate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555541716
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/SegmentedLruPageList.java
##########
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Pages Segmented-LRU (SLRU) list implementation.
+ *
+ * @see PageReplacementMode#SEGMENTED_LRU
+ */
+public class SegmentedLruPageList {
+ /** Ratio to limit count of protected pages. */
+ private static final double PROTECTED_TO_TOTAL_PAGES_RATIO = 0.5;
+
+ /** Null page index. */
+ static final int NULL_IDX = -1;
+
+ /** Index of the head page of LRU list. */
+ private int headIdx = NULL_IDX;
+
+ /** Index of the tail page of LRU list. */
+ private int tailIdx = NULL_IDX;
+
+ /** Index of the tail page of probationary segment. */
+ private int probTailIdx = NULL_IDX;
+
+ /** Count of protected pages in the list. */
+ private int protectedPagesCnt;
+
+ /** Protected pages segment limit. */
+ private final int protectedPagesLimit;
+
+ /** Pointer to memory region to store links. */
+ private final long linksPtr;
+
+ /** Pointer to memory region to store protected flags. */
+ private final long flagsPtr;
+
+ /**
+ * @param totalPagesCnt Total pages count.
+ * @param memPtr Pointer to memory region.
+ */
+ public SegmentedLruPageList(int totalPagesCnt, long memPtr) {
+ linksPtr = memPtr;
+ flagsPtr = memPtr + (((long)totalPagesCnt) << 3);
+
+ GridUnsafe.setMemory(linksPtr, ((long)totalPagesCnt) << 3, (byte)0xFF);
+ GridUnsafe.setMemory(flagsPtr, (totalPagesCnt + 7) >> 3, (byte)0);
+
+ protectedPagesLimit = (int)(totalPagesCnt * PROTECTED_TO_TOTAL_PAGES_RATIO);
+ }
+
+ /**
+ * Remove page from the head of LRU list.
+ *
+ * @return Page index or {@code -1} if list is empty.
+ */
+ public synchronized int poll() {
+ int idx = headIdx;
+
+ if (idx != NULL_IDX)
+ remove(idx);
+
+ return idx;
+ }
+
+ /**
+ * Remove page from LRU list by page index.
+ *
+ * @param pageIdx Page index.
+ */
+ public synchronized void remove(int pageIdx) {
Review comment:
this methoad are always calling under wlock as i see, thus no need in sync here, only wlock hold check or simple comment, i hope.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/SegmentedLruPageList.java
##########
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Pages Segmented-LRU (SLRU) list implementation.
+ *
+ * @see PageReplacementMode#SEGMENTED_LRU
+ */
+public class SegmentedLruPageList {
+ /** Ratio to limit count of protected pages. */
+ private static final double PROTECTED_TO_TOTAL_PAGES_RATIO = 0.5;
+
+ /** Null page index. */
+ static final int NULL_IDX = -1;
+
+ /** Index of the head page of LRU list. */
+ private int headIdx = NULL_IDX;
+
+ /** Index of the tail page of LRU list. */
+ private int tailIdx = NULL_IDX;
+
+ /** Index of the tail page of probationary segment. */
+ private int probTailIdx = NULL_IDX;
+
+ /** Count of protected pages in the list. */
+ private int protectedPagesCnt;
+
+ /** Protected pages segment limit. */
+ private final int protectedPagesLimit;
+
+ /** Pointer to memory region to store links. */
+ private final long linksPtr;
+
+ /** Pointer to memory region to store protected flags. */
+ private final long flagsPtr;
+
+ /**
+ * @param totalPagesCnt Total pages count.
+ * @param memPtr Pointer to memory region.
+ */
+ public SegmentedLruPageList(int totalPagesCnt, long memPtr) {
+ linksPtr = memPtr;
+ flagsPtr = memPtr + (((long)totalPagesCnt) << 3);
+
+ GridUnsafe.setMemory(linksPtr, ((long)totalPagesCnt) << 3, (byte)0xFF);
+ GridUnsafe.setMemory(flagsPtr, (totalPagesCnt + 7) >> 3, (byte)0);
+
+ protectedPagesLimit = (int)(totalPagesCnt * PROTECTED_TO_TOTAL_PAGES_RATIO);
+ }
+
+ /**
+ * Remove page from the head of LRU list.
+ *
+ * @return Page index or {@code -1} if list is empty.
+ */
+ public synchronized int poll() {
+ int idx = headIdx;
+
+ if (idx != NULL_IDX)
+ remove(idx);
+
+ return idx;
+ }
+
+ /**
+ * Remove page from LRU list by page index.
+ *
+ * @param pageIdx Page index.
+ */
+ public synchronized void remove(int pageIdx) {
Review comment:
this method are always calling under wlock as i see, thus no need in sync here, only wlock hold check or simple comment, i hope.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r556273370
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/ClockPageReplacementFlags.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import java.util.function.LongUnaryOperator;
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Clock page replacement algorithm implementation.
+ *
+ * @see PageReplacementMode#CLOCK
+ */
+public class ClockPageReplacementFlags {
+ /** Total pages count. */
+ private final int pagesCnt;
+
+ /** Index of the next candidate ("hand"). */
+ private volatile int curIdx;
Review comment:
if so - can you remove it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555006222
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/ClockPageReplacementFlags.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import java.util.function.LongUnaryOperator;
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Clock page replacement algorithm implementation.
+ *
+ * @see PageReplacementMode#CLOCK
+ */
+public class ClockPageReplacementFlags {
+ /** Total pages count. */
+ private final int pagesCnt;
+
+ /** Index of the next candidate ("hand"). */
+ private volatile int curIdx;
Review comment:
why do we need volatile here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r554986017
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/PageReplacementMode.java
##########
@@ -0,0 +1,85 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.ignite.configuration;
+
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Defines memory page replacement algorithm. A mode is set for a specific {@link DataRegionConfiguration}.
+ */
+public enum PageReplacementMode {
+ /**
+ * Random-LRU algorithm.
+ *
+ * Every time page is accessed, its timestamp gets updated. When a page fault occurs and it's required to replace
+ * some pages, the algorithm randomly chooses 5 pages from the page memory and evicts a page with the latest
+ * timestamp.
+ *
+ * This algorithm have zero maintenance cost, but not very effective in terms of finding next page to replace.
+ * Recommended to use in environments, where there are no page replacements (large enough data region to store all
Review comment:
How engineer or someone who would read this can detect is there are page replacements at all ? I suggest to add metric link here ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555061043
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/PageReplacementMode.java
##########
@@ -0,0 +1,85 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.ignite.configuration;
+
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Defines memory page replacement algorithm. A mode is set for a specific {@link DataRegionConfiguration}.
+ */
+public enum PageReplacementMode {
+ /**
+ * Random-LRU algorithm.
+ *
+ * Every time page is accessed, its timestamp gets updated. When a page fault occurs and it's required to replace
+ * some pages, the algorithm randomly chooses 5 pages from the page memory and evicts a page with the latest
Review comment:
`RandomLruPageReplacementPolicy#RANDOM_PAGES_EVICT_NUM` is in internal package. Generated JavaDocs for AI release will be broken.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555056874
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/ClockPageReplacementFlags.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import java.util.function.LongUnaryOperator;
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Clock page replacement algorithm implementation.
+ *
+ * @see PageReplacementMode#CLOCK
+ */
+public class ClockPageReplacementFlags {
+ /** Total pages count. */
+ private final int pagesCnt;
+
+ /** Index of the next candidate ("hand"). */
+ private volatile int curIdx;
Review comment:
Looks like it's not necessary, since curIdx is always modified under segment write lock and HB provided by this lock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r554984594
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/PageReplacementMode.java
##########
@@ -0,0 +1,85 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.ignite.configuration;
+
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Defines memory page replacement algorithm. A mode is set for a specific {@link DataRegionConfiguration}.
+ */
+public enum PageReplacementMode {
+ /**
+ * Random-LRU algorithm.
+ *
+ * Every time page is accessed, its timestamp gets updated. When a page fault occurs and it's required to replace
+ * some pages, the algorithm randomly chooses 5 pages from the page memory and evicts a page with the latest
Review comment:
seems hardcoded "5" need to be replaced with {@link RandomLruPageReplacementPolicy#RANDOM_PAGES_EVICT_NUM} here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] zstan commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
zstan commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r554995246
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/ClockPageReplacementFlags.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import java.util.function.LongUnaryOperator;
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Clock page replacement algorithm implementation.
+ *
+ * @see PageReplacementMode#CLOCK
+ */
+public class ClockPageReplacementFlags {
+ /** Total pages count. */
+ private final int pagesCnt;
+
+ /** Index of the next candidate ("hand"). */
+ private volatile int curIdx;
+
+ /** Pointer to memory region to store page hit flags. */
+ private final long flagsPtr;
+
+ /**
+ * @param totalPagesCnt Total pages count.
+ * @param memPtr Pointer to memory region.
+ */
+ ClockPageReplacementFlags(int totalPagesCnt, long memPtr) {
+ pagesCnt = totalPagesCnt;
+ flagsPtr = memPtr;
+
+ GridUnsafe.setMemory(flagsPtr, (totalPagesCnt + 7) >> 3, (byte)0);
+ }
+
+ /**
+ * Find page to replace.
+ *
+ * @return Page index to replace.
+ */
+ @SuppressWarnings("NonAtomicOperationOnVolatileField")
+ public int poll() {
+ // This method is always executed under exclusive lock, no other synchronization or CAS required.
Review comment:
probably it would be helpful to add wlock holds by current thread assertion here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555058028
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/ClockPageReplacementFlags.java
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import java.util.function.LongUnaryOperator;
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Clock page replacement algorithm implementation.
+ *
+ * @see PageReplacementMode#CLOCK
+ */
+public class ClockPageReplacementFlags {
+ /** Total pages count. */
+ private final int pagesCnt;
+
+ /** Index of the next candidate ("hand"). */
+ private volatile int curIdx;
+
+ /** Pointer to memory region to store page hit flags. */
+ private final long flagsPtr;
+
+ /**
+ * @param totalPagesCnt Total pages count.
+ * @param memPtr Pointer to memory region.
+ */
+ ClockPageReplacementFlags(int totalPagesCnt, long memPtr) {
+ pagesCnt = totalPagesCnt;
+ flagsPtr = memPtr;
+
+ GridUnsafe.setMemory(flagsPtr, (totalPagesCnt + 7) >> 3, (byte)0);
+ }
+
+ /**
+ * Find page to replace.
+ *
+ * @return Page index to replace.
+ */
+ @SuppressWarnings("NonAtomicOperationOnVolatileField")
+ public int poll() {
+ // This method is always executed under exclusive lock, no other synchronization or CAS required.
Review comment:
We don't have the segment write lock in this class. I think it's not such a good idea to pass it and store it here only for assertion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] alex-plekhanov commented on a change in pull request #8513: IGNITE-13761 Segmented-LRU and CLOCK page replacement
Posted by GitBox <gi...@apache.org>.
alex-plekhanov commented on a change in pull request #8513:
URL: https://github.com/apache/ignite/pull/8513#discussion_r555544359
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/SegmentedLruPageList.java
##########
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.persistence.pagemem;
+
+import org.apache.ignite.configuration.PageReplacementMode;
+import org.apache.ignite.internal.util.GridUnsafe;
+
+/**
+ * Pages Segmented-LRU (SLRU) list implementation.
+ *
+ * @see PageReplacementMode#SEGMENTED_LRU
+ */
+public class SegmentedLruPageList {
+ /** Ratio to limit count of protected pages. */
+ private static final double PROTECTED_TO_TOTAL_PAGES_RATIO = 0.5;
+
+ /** Null page index. */
+ static final int NULL_IDX = -1;
+
+ /** Index of the head page of LRU list. */
+ private int headIdx = NULL_IDX;
+
+ /** Index of the tail page of LRU list. */
+ private int tailIdx = NULL_IDX;
+
+ /** Index of the tail page of probationary segment. */
+ private int probTailIdx = NULL_IDX;
+
+ /** Count of protected pages in the list. */
+ private int protectedPagesCnt;
+
+ /** Protected pages segment limit. */
+ private final int protectedPagesLimit;
+
+ /** Pointer to memory region to store links. */
+ private final long linksPtr;
+
+ /** Pointer to memory region to store protected flags. */
+ private final long flagsPtr;
+
+ /**
+ * @param totalPagesCnt Total pages count.
+ * @param memPtr Pointer to memory region.
+ */
+ public SegmentedLruPageList(int totalPagesCnt, long memPtr) {
+ linksPtr = memPtr;
+ flagsPtr = memPtr + (((long)totalPagesCnt) << 3);
+
+ GridUnsafe.setMemory(linksPtr, ((long)totalPagesCnt) << 3, (byte)0xFF);
+ GridUnsafe.setMemory(flagsPtr, (totalPagesCnt + 7) >> 3, (byte)0);
+
+ protectedPagesLimit = (int)(totalPagesCnt * PROTECTED_TO_TOTAL_PAGES_RATIO);
+ }
+
+ /**
+ * Remove page from the head of LRU list.
+ *
+ * @return Page index or {@code -1} if list is empty.
+ */
+ public synchronized int poll() {
+ int idx = headIdx;
+
+ if (idx != NULL_IDX)
+ remove(idx);
+
+ return idx;
+ }
+
+ /**
+ * Remove page from LRU list by page index.
+ *
+ * @param pageIdx Page index.
+ */
+ public synchronized void remove(int pageIdx) {
Review comment:
It's not under write lock in `ClearSegmentRunnable`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org