You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/11/17 14:28:02 UTC
[GitHub] [ignite] sergey-chugunov-1985 opened a new pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
sergey-chugunov-1985 opened a new pull request #8465:
URL: https://github.com/apache/ignite/pull/8465
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558336380
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
Review comment:
Please use a code block for the command here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r560209238
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
Review comment:
I found out that historical rebalancing isn't covered with documentation yet so I removed mention of the link.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558330837
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
Review comment:
@sergey-chugunov-1985 Please add a link to the **Historical rebalance** page here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 closed pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 closed pull request #8465:
URL: https://github.com/apache/ignite/pull/8465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558336506
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
+====
+
+When node executes defragmentation it is possible to cancel it using the following command available in control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
Review comment:
Please use a code block for the command here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Mmuzaf commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Mmuzaf commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764612971
@sergey-chugunov-1985 @Nikita-tech-writer
Thank you for your efforts. I'll merge the issue when the travis checks pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558337391
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
Review comment:
```suggestion
After the manual restart, the node with the requested defragmentation enters a special mode called maintenance mode. The node in maintenance mode does not join the rest of the cluster but remains isolated until defragmentation is completed (or canceled by explicit user request). After that, the user has to restart the node one more time: it exits maintenance mode and returns to normal operations (joining the cluster and starting to serve regular workload).
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558341820
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
+====
+
+When node executes defragmentation it is possible to cancel it using the following command available in control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
+
+For more information about commands refer to their help.
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation caches are defragmented one by one (if defragmentation of more than one cache was requested). To calculate additional space required find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
+====
+
+== Conclusion
+In most situations defragmentation isn't needed as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases it may be necessary to employ it to free up disk space.
Review comment:
```suggestion
In most situations, defragmentation is not necessary as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases, it may be necessary to free disk space up.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558339263
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
+====
+
+When node executes defragmentation it is possible to cancel it using the following command available in control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
+
+For more information about commands refer to their help.
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558332970
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
Review comment:
```suggestion
== Performing Defragmentation
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764571959
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558334972
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
Review comment:
```suggestion
Defragmentation is a costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r561312247
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
+
+=== Starting Defragmentation
+
+To request defragmentation use the following command:
+[source,shell]
+----
+control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]
+----
+
+After the manual restart, the node with the requested defragmentation enters a special mode called maintenance mode. The node in maintenance mode does not join the rest of the cluster but remains isolated until defragmentation is completed (or canceled by explicit user request). After that, the user has to restart the node one more time: it exits maintenance mode and returns to normal operations (joining the cluster and starting to serve regular workload).
+
+[NOTE]
+====
+Nodes in maintenance mode do not participate in serving the regular workload. It is not recommended to execute defragmentation on several nodes simultaneously as it reduces the number of backups, thus increasing the risk of partition loss.
+====
+
+=== Stopping Defragmentation
+
+When a node executes defragmentation, it is possible to cancel it. To stop defragmentation, run the following command available in the control utility:
+[source,shell]
+----
+control.(sh|bat) --defragmentation cancel --host --port
+----
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation, caches are defragmented one by one (if a defragmentation of more than one cache is requested). To calculate additional required space, find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
+====
+
+== Conclusion
+In most situations defragmentation isn't needed as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases it may be necessary to employ it to free up disk space.
+
+Its usage requires taking nodes out of normal operations so it careful planning is needed.
Review comment:
```suggestion
Persistence defragmentation requires taking nodes out of their normal operations, so a careful planning is recommended.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558334972
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
Review comment:
```suggestion
Defragmentation is a costly operation in terms of disk IO. To avoid slowing down user operations, do not execute it on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r560157802
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
Review comment:
Disagree on the suggested change: "do not execute" implies that it is possible to do the action but is not recommended. In reality executing defragmentation on a regular node is not possible at all. How can we rephrase this to catch this meaning?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558335816
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
Review comment:
```suggestion
=== Starting Defragmentation
To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558327938
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
Review comment:
```suggestion
In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558337918
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
Review comment:
```suggestion
Nodes in maintenance mode do not participate in serving the regular workload. It is not recommended to execute defragmentation on several nodes simultaneously as it reduces the number of backups, thus increasing the risk of partition loss.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764601197
@Mmuzaf I suppouse this PR can be merged now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r561310059
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
+
+=== Starting Defragmentation
+
+To request defragmentation use the following command:
Review comment:
```suggestion
To request defragmentation, use the following command:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764571959
> @sergey-chugunov-1985
>
> Do we need something to resolve the comments? Can we proceed with this PR?
@Mmuzaf I hope I'm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764601197
@Mmuzaf I suppouse this PR can be merged now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r561308961
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
Review comment:
```suggestion
Defragmentation enables a user to shrink data files and claim back disk space.
```
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
Review comment:
```suggestion
Defragmentation is a costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
```
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
+
+=== Starting Defragmentation
+
+To request defragmentation use the following command:
Review comment:
```suggestion
To request defragmentation, use the following command:
```
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
+
+=== Starting Defragmentation
+
+To request defragmentation use the following command:
+[source,shell]
+----
+control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]
+----
+
+After the manual restart, the node with the requested defragmentation enters a special mode called maintenance mode. The node in maintenance mode does not join the rest of the cluster but remains isolated until defragmentation is completed (or canceled by explicit user request). After that, the user has to restart the node one more time: it exits maintenance mode and returns to normal operations (joining the cluster and starting to serve regular workload).
+
+[NOTE]
+====
+Nodes in maintenance mode do not participate in serving the regular workload. It is not recommended to execute defragmentation on several nodes simultaneously as it reduces the number of backups, thus increasing the risk of partition loss.
+====
+
+=== Stopping Defragmentation
+
+When a node executes defragmentation, it is possible to cancel it. To stop defragmentation, run the following command available in the control utility:
+[source,shell]
+----
+control.(sh|bat) --defragmentation cancel --host --port
+----
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation, caches are defragmented one by one (if a defragmentation of more than one cache is requested). To calculate additional required space, find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
+====
+
+== Conclusion
+In most situations defragmentation isn't needed as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases it may be necessary to employ it to free up disk space.
Review comment:
```suggestion
In most situations, defragmentation is not necessary as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases, it may be necessary to free disk space up.
```
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
+
+=== Starting Defragmentation
+
+To request defragmentation use the following command:
+[source,shell]
+----
+control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]
+----
+
+After the manual restart, the node with the requested defragmentation enters a special mode called maintenance mode. The node in maintenance mode does not join the rest of the cluster but remains isolated until defragmentation is completed (or canceled by explicit user request). After that, the user has to restart the node one more time: it exits maintenance mode and returns to normal operations (joining the cluster and starting to serve regular workload).
+
+[NOTE]
+====
+Nodes in maintenance mode do not participate in serving the regular workload. It is not recommended to execute defragmentation on several nodes simultaneously as it reduces the number of backups, thus increasing the risk of partition loss.
+====
+
+=== Stopping Defragmentation
+
+When a node executes defragmentation, it is possible to cancel it. To stop defragmentation, run the following command available in the control utility:
+[source,shell]
+----
+control.(sh|bat) --defragmentation cancel --host --port
+----
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation, caches are defragmented one by one (if a defragmentation of more than one cache is requested). To calculate additional required space, find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
+====
+
+== Conclusion
+In most situations defragmentation isn't needed as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases it may be necessary to employ it to free up disk space.
+
+Its usage requires taking nodes out of normal operations so it careful planning is needed.
Review comment:
```suggestion
Persistence defragmentation requires taking nodes out of their normal operations, so a careful planning is recommended.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r561308961
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
Review comment:
```suggestion
Defragmentation enables a user to shrink data files and claim back disk space.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558340756
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
+====
+
+When node executes defragmentation it is possible to cancel it using the following command available in control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
+
+For more information about commands refer to their help.
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation caches are defragmented one by one (if defragmentation of more than one cache was requested). To calculate additional space required find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
Review comment:
```suggestion
To reduce disk space requirements during defragmentation, caches are defragmented one by one (if a defragmentation of more than one cache is requested). To calculate additional required space, find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558326083
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
Review comment:
```suggestion
Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764573007
> @sergey-chugunov-1985
>
> Do we need something to resolve the comments? Can we proceed with this PR?
@Mmuzaf I addressed (as I hope) all comments to the PR, it should be ready to merge soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558327938
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
Review comment:
```suggestion
In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r560239589
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
Review comment:
Then, lets leave the first version with minor changes (I've also updated the suggested change):
Defragmentation is a costly operation in terms of disk IO. **To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster.** To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
Review comment:
Then, let's leave the first version with minor changes (I've also updated the suggested change):
Defragmentation is a costly operation in terms of disk IO. **To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster.** To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558331426
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558329984
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
Review comment:
```suggestion
Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 removed a comment on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 removed a comment on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764571959
> @sergey-chugunov-1985
>
> Do we need something to resolve the comments? Can we proceed with this PR?
@Mmuzaf I hope I'm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r561311729
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
+
+=== Starting Defragmentation
+
+To request defragmentation use the following command:
+[source,shell]
+----
+control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]
+----
+
+After the manual restart, the node with the requested defragmentation enters a special mode called maintenance mode. The node in maintenance mode does not join the rest of the cluster but remains isolated until defragmentation is completed (or canceled by explicit user request). After that, the user has to restart the node one more time: it exits maintenance mode and returns to normal operations (joining the cluster and starting to serve regular workload).
+
+[NOTE]
+====
+Nodes in maintenance mode do not participate in serving the regular workload. It is not recommended to execute defragmentation on several nodes simultaneously as it reduces the number of backups, thus increasing the risk of partition loss.
+====
+
+=== Stopping Defragmentation
+
+When a node executes defragmentation, it is possible to cancel it. To stop defragmentation, run the following command available in the control utility:
+[source,shell]
+----
+control.(sh|bat) --defragmentation cancel --host --port
+----
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation, caches are defragmented one by one (if a defragmentation of more than one cache is requested). To calculate additional required space, find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
+====
+
+== Conclusion
+In most situations defragmentation isn't needed as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases it may be necessary to employ it to free up disk space.
Review comment:
```suggestion
In most situations, defragmentation is not necessary as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases, it may be necessary to free disk space up.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Mmuzaf commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Mmuzaf commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764612971
@sergey-chugunov-1985 @Nikita-tech-writer
Thank you for your efforts. I'll merge the issue when the travis checks pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 closed pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 closed pull request #8465:
URL: https://github.com/apache/ignite/pull/8465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558343587
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
+====
+
+When node executes defragmentation it is possible to cancel it using the following command available in control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
+
+For more information about commands refer to their help.
+
+[NOTE]
+====
+To reduce disk space requirements during defragmentation caches are defragmented one by one (if defragmentation of more than one cache was requested). To calculate additional space required find the cache that occupies the most disk space. The same amount of disk space is required for defragmentation at max.
+====
+
+== Conclusion
+In most situations defragmentation isn't needed as existing memory management mechanism effectively reuses memory left after data deletion. But in rare cases it may be necessary to employ it to free up disk space.
+
+Its usage requires taking nodes out of normal operations so it careful planning is needed.
Review comment:
```suggestion
Persistence defragmentation requires taking nodes out of their normal operations, so a careful planning is recommended.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Mmuzaf merged pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Mmuzaf merged pull request #8465:
URL: https://github.com/apache/ignite/pull/8465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Mmuzaf merged pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Mmuzaf merged pull request #8465:
URL: https://github.com/apache/ignite/pull/8465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558330837
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
Review comment:
@sergey-chugunov-1985 Please add a link to the **Historical rebalance** page.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r561309918
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So the files, where Ignite persists data, can only grow and never shrink.
+
+In most use cases, it does not cause any problem as once created page can be reused multiple times. However, in certain cases, it is possible that cache contains very little data but occupies large chunks of disk space because a significant volume of data was removed from the cache.
+
+Defragmentation enables user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled. If historical rebalancing is disabled, the server node always triggers full rebalance after the restart, which would throw away the defragmented partition. A full set of data is transferred to the node from other nodes over a network. Depending on the dataset’s size, transferring may require significant time and slow down the whole cluster as network capacity is essential to fulfill user requests.
+====
+
+== Performing Defragmentation
+
+Defragmentation is costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
Review comment:
```suggestion
Defragmentation is a costly operation in terms of disk IO. To avoid slowing down user operations, note that defragmentation cannot be executed on a regular node joined to the cluster. To perform defragmentation, you need to request it first on a particular node or set of nodes and then restart them.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-762397635
@sergey-chugunov-1985 @Mmuzaf
We should definitely apply the changes I suggested and then proceed with the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558338799
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
+
+[NOTE]
+====
+Defragmentation can only be used with historical rebalance enabled (link to historical rebalance page). If historical rebalance is disabled server node always triggers full rebalance after restart throwing away defragmented partition. Full set of data is transferred to the node from other nodes over network, depending of size of data set it may require a lot of time and may slow down the whole cluster as network capacity is important to fulfill user requests.
+====
+
+== Usage: how and when
+
+Defragmentation is costly operation in terms of disk IO so to avoid slowing down user operations it cannot be executed on regular node joined to the cluster. To execute defragmentation user needs to request it first on a particular node or set of nodes and than restart these nodes.
+
+To request defragmentation use the following command: *control.(sh|bat) --defragmentation schedule --nodes <consistentIds> [--caches <cacheNames>]*.
+
+After restart node with requested defragmentation will enter special mode called maintenance mode. Node in maintenance doesn't join the rest of the cluster but stays isolated until defragmentation is completed (or cancelled by explicit user request). After that user has to restart the node one more time: it will exit maintenance mode and returns back to normal operations (joins the cluster and starts to serve regular workload).
+
+[NOTE]
+====
+As nodes in maintenance don't participate in serving usual workload, it is not recommended to execute defragmentation on several nodes at once as it reduces number of backups thus increasing the risk of partition loss.
+====
+
+When node executes defragmentation it is possible to cancel it using the following command available in control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
Review comment:
```suggestion
=== Stopping Defragmentation
When a node executes defragmentation, it is possible to cancel it. To stop defragmentation, run the following command available in the control utility: *control.(sh|bat) --defragmentation cancel --host --port.*
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 removed a comment on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 removed a comment on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-764571959
> @sergey-chugunov-1985
>
> Do we need something to resolve the comments? Can we proceed with this PR?
@Mmuzaf I hope I'm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558328636
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
+
+In most use cases it doesn't cause any problems as once created page can be reused multiple times. However in certain cases it is possible that cache contains very little data but occupies large chunks of disk space because a lot of data was removed from the cache.
+
+Defragmentation is aimed to enable user to shrink data files and claim back disk space.
Review comment:
```suggestion
Defragmentation enables user to shrink data files and claim back disk space.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Mmuzaf commented on pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Mmuzaf commented on pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#issuecomment-762231433
@sergey-chugunov-1985
Do we need something to resolve the comments? Can we proceed with this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Nikita-tech-writer commented on a change in pull request #8465: IGNITE-13674 Documentation page added for Native Persistence Defragmentation feature
Posted by GitBox <gi...@apache.org>.
Nikita-tech-writer commented on a change in pull request #8465:
URL: https://github.com/apache/ignite/pull/8465#discussion_r558326083
##########
File path: docs/_docs/persistence/native-persistence-defragmentation.adoc
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+= Persistence Defragmentation
+
+== Overview
+
+As memory management mechanism of Apache Ignite can only create or reuse pages for user data but never frees them files where Ignite persists data can only grow and never shrinks.
Review comment:
```suggestion
Apache Ignite memory management mechanism can only create or reuse pages for user data, but it never frees them. So files, where Ignite persists data, can only grow, but never shrink.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org