You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/09/22 14:13:00 UTC
[GitHub] [ignite] Sega76 opened a new pull request #8267: IGNITE-13473: Snapshot test for ducktape
Sega76 opened a new pull request #8267:
URL: https://github.com/apache/ignite/pull/8267
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494264375
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494142676
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
dump3 is equal to dump1
because we restored from a snapshot and the data in the cluster is the identical
idle_verify --dump
generates a file that contains itself counters and data hash sums
Partition: PartitionKeyV2 [grpId=623628935, grpName=test-cache, partId=0]
Partition instances: [PartitionHashRecordV2 [isPrimary=true, consistentId=ducker03, updateCntr=73, partitionState=OWNING, size=73, partHash=-878633494]]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#issuecomment-701342329
created new PR https://github.com/apache/ignite/pull/8294
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r495806691
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -30,13 +30,20 @@
@Override public void run(JsonNode jsonNode) {
IgniteCache<UUID, UUID> cache = ignite.getOrCreateCache(jsonNode.get("cacheName").asText());
+ JsonNode jNodeSize = jsonNode.get("size");
+
+ long size = -1L;
+
long cnt = 0L;
+ if (jNodeSize != null)
+ size = jNodeSize.asLong();
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494178592
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
I believe the less a test runs the better. Why do not use only 1 sec? Or can we provide a counter limit or time limit to the application itself. After counter has exceed the application just stops. And we can continue test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494208588
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
Unfortunatelly, ducktape allows start node only with cleaning. We should add smth like restart and comment that this should be rewrittten after appropriate change in ducktape codebase will be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494180548
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
Don't like it. Ducktape Service provides `start` method and it do some additional work than `start_node`. It is misleading which method (run or start) to invoke. As you add rotate log to start, can you remove run method at all?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494166786
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,49 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_workdir(self):
+ """
+ Copy work dir from service nodes to the results directory.
+
+ If the test passed, only the default set will be collected. If the the test failed, all logs will be collected.
+ """
+ self.logger.info("copy_ignite_workdir")
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.test_context.logger.debug("Won't collect service workdir from %s." %
+ service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the service logs
+ self.test_context.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.test_context.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_work = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_work))
+ node.account.copy_from(tgz_work, dest)
+ except Exception as e:
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494163284
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ *
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494163539
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -136,6 +158,35 @@ def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("sudo rm -rf -- %s" % self.PERSISTENT_ROOT, allow_fail=False)
+ def rename_db(self, new_db_name: str):
+ """
+ Rename db.
+ """
+ for node in self.nodes:
+ self._rename_db(node, new_db_name)
+
+ def _rename_db(self, node, new_db_name: str):
+ """
+ Rename db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo mv {self.WORK_DIR}/db {self.WORK_DIR}/{new_db_name}", allow_fail=False)
+
+ def copy_snap_to_db(self, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ for node in self.nodes:
+ self._copy_snap_to_db(node, snapshot_name)
+
+ def _copy_snap_to_db(self, node, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo cp -r {self.SNAPSHOT}/{snapshot_name}/db {self.WORK_DIR}", allow_fail=False)
+ node.account.ssh(f"sudo chown -R ducker:ducker {self.PERSISTENT_ROOT}", allow_fail=False)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r493832026
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,49 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_workdir(self):
+ """
+ Copy work dir from service nodes to the results directory.
+
+ If the test passed, only the default set will be collected. If the the test failed, all logs will be collected.
+ """
+ self.logger.info("copy_ignite_workdir")
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.test_context.logger.debug("Won't collect service workdir from %s." %
+ service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the service logs
+ self.test_context.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.test_context.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_work = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_work))
+ node.account.copy_from(tgz_work, dest)
+ except Exception as e:
Review comment:
You should use `ex` instead of `e`, Disable warning of broad exception using inline comment:
```
except Exception as ex: # pylint: disable=W0703
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494166560
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -16,12 +16,16 @@
"""
This module contains basic ignite test.
"""
+import os
from time import monotonic
-from ducktape.tests.test import Test
+from ducktape.utils.local_filesystem_utils import mkdir_p
+from ducktape.tests.test import Test, TestContext
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.utils.ignite_persistence import PersistenceAware, IgnitePersistenceAware
-# pylint: disable=W0223
+# pylint: disable=C0103,W0223,W0703
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r495814886
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -78,7 +77,7 @@ def snapshot_test(self, ignite_version):
self.test_context,
client_config,
java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
- params={"cacheName": "test-cache"})
+ params={"cacheName": "test-cache", "size": 1024})
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r493407882
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ *
Review comment:
Please add meaningful java doc
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
Review comment:
Type, "ю" instead of "."
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
Am I correct that equality of dump1 and dump3 is achieved by the fact that streamer loads data the same time (1 min long for every load)?
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -136,6 +158,35 @@ def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("sudo rm -rf -- %s" % self.PERSISTENT_ROOT, allow_fail=False)
+ def rename_db(self, new_db_name: str):
+ """
+ Rename db.
+ """
+ for node in self.nodes:
+ self._rename_db(node, new_db_name)
+
+ def _rename_db(self, node, new_db_name: str):
+ """
+ Rename db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo mv {self.WORK_DIR}/db {self.WORK_DIR}/{new_db_name}", allow_fail=False)
+
+ def copy_snap_to_db(self, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ for node in self.nodes:
+ self._copy_snap_to_db(node, snapshot_name)
+
+ def _copy_snap_to_db(self, node, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo cp -r {self.SNAPSHOT}/{snapshot_name}/db {self.WORK_DIR}", allow_fail=False)
+ node.account.ssh(f"sudo chown -R ducker:ducker {self.PERSISTENT_ROOT}", allow_fail=False)
Review comment:
Why do you use sudo? There are chances that this operation will be forbidden on test cluster. Looks like this ownership must exist on cluster before tests run
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
Review comment:
Type, miss "." in the end of line.
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
Is it a real need to load 1 minute long?
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ *
+ */
+public class UuidStreamerApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jsonNode) {
+ IgniteCache<UUID, UUID> cache = ignite.getOrCreateCache(jsonNode.get("cacheName").asText());
+
+ AtomicInteger counter = new AtomicInteger();
Review comment:
Why do you use atomic for local variable?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
Review comment:
move logging data inside `snapshot_create` (as this is the only reason for defining `data` variable) and move it to DEBUG.
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -16,12 +16,16 @@
"""
This module contains basic ignite test.
"""
+import os
from time import monotonic
-from ducktape.tests.test import Test
+from ducktape.utils.local_filesystem_utils import mkdir_p
+from ducktape.tests.test import Test, TestContext
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.utils.ignite_persistence import PersistenceAware, IgnitePersistenceAware
-# pylint: disable=W0223
+# pylint: disable=C0103,W0223,W0703
Review comment:
C0103 "Used when a name doesn't doesn't fit the naming convention associated to its type". Why do you disable it?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
Ah, ok, I see.
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
I believe the less a test runs the better. Why do not use only 1 sec? Or can we provide a counter limit or time limit to the application itself. After counter has exceed the application just stops. And we can continue test.
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
Don't like it. Ducktape Service provides `start` method and it do some additional work than `start_node`. It is misleading which method (run or start) to invoke. As you add rotate log to start, can you remove run method at all?
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
Restart method with doc is ok. Let's do it this way.
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
Why do you think so? I think there should be whether amount data that matters (depends on config, internal implementation or known issues) or minimal data to run a test. It can be dozens or hundreds puts because it fast, but just wait 2 mins without explanation isn't an option.
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -30,13 +30,20 @@
@Override public void run(JsonNode jsonNode) {
IgniteCache<UUID, UUID> cache = ignite.getOrCreateCache(jsonNode.get("cacheName").asText());
+ JsonNode jNodeSize = jsonNode.get("size");
+
+ long size = -1L;
+
long cnt = 0L;
+ if (jNodeSize != null)
+ size = jNodeSize.asLong();
Review comment:
long size = Optional.ofNullable(jNode.get("size")).map(JsonNode::asLong).orElse(-1L);
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -78,7 +77,7 @@ def snapshot_test(self, ignite_version):
self.test_context,
client_config,
java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
- params={"cacheName": "test-cache"})
+ params={"cacheName": "test-cache", "size": 1024})
Review comment:
Is 1024 enough to fill all partitions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494163775
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,49 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_workdir(self):
+ """
+ Copy work dir from service nodes to the results directory.
+
+ If the test passed, only the default set will be collected. If the the test failed, all logs will be collected.
+ """
+ self.logger.info("copy_ignite_workdir")
+ for service in self.test_context.services:
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494960091
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -30,13 +30,20 @@
@Override public void run(JsonNode jsonNode) {
IgniteCache<UUID, UUID> cache = ignite.getOrCreateCache(jsonNode.get("cacheName").asText());
+ JsonNode jNodeSize = jsonNode.get("size");
+
+ long size = -1L;
+
long cnt = 0L;
+ if (jNodeSize != null)
+ size = jNodeSize.asLong();
Review comment:
long size = Optional.ofNullable(jNode.get("size")).map(JsonNode::asLong).orElse(-1L);
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494960405
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -78,7 +77,7 @@ def snapshot_test(self, ignite_version):
self.test_context,
client_config,
java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
- params={"cacheName": "test-cache"})
+ params={"cacheName": "test-cache", "size": 1024})
Review comment:
Is 1024 enough to fill all partitions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494127689
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
No, this is because snapshot is restored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494146870
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
Perhaps less or more, i use a minute for the test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494176192
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
Ah, ok, I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494291109
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
Why do you think so? I think there should be whether amount data that matters (depends on config, internal implementation or known issues) or minimal data to run a test. It can be dozens or hundreds puts because it fast, but just wait 2 mins without explanation isn't an option.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494142676
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
dump3 is equal to dump1
because we restored from a snapshot and the data in the cluster is the identical
idle_verify --dump
generates a file that contains itself counters and data hash sums
Partition: PartitionKeyV2 [grpId=623628935, grpName=test-cache, partId=0]
Partition instances: [PartitionHashRecordV2 [isPrimary=true, consistentId=ducker03, updateCntr=73, partitionState=OWNING, size=73, partHash=-878633494]]
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
Perhaps less or more, i use a minute for the test
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ *
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -136,6 +158,35 @@ def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("sudo rm -rf -- %s" % self.PERSISTENT_ROOT, allow_fail=False)
+ def rename_db(self, new_db_name: str):
+ """
+ Rename db.
+ """
+ for node in self.nodes:
+ self._rename_db(node, new_db_name)
+
+ def _rename_db(self, node, new_db_name: str):
+ """
+ Rename db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo mv {self.WORK_DIR}/db {self.WORK_DIR}/{new_db_name}", allow_fail=False)
+
+ def copy_snap_to_db(self, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ for node in self.nodes:
+ self._copy_snap_to_db(node, snapshot_name)
+
+ def _copy_snap_to_db(self, node, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo cp -r {self.SNAPSHOT}/{snapshot_name}/db {self.WORK_DIR}", allow_fail=False)
+ node.account.ssh(f"sudo chown -R ducker:ducker {self.PERSISTENT_ROOT}", allow_fail=False)
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,49 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_workdir(self):
+ """
+ Copy work dir from service nodes to the results directory.
+
+ If the test passed, only the default set will be collected. If the the test failed, all logs will be collected.
+ """
+ self.logger.info("copy_ignite_workdir")
+ for service in self.test_context.services:
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -16,12 +16,16 @@
"""
This module contains basic ignite test.
"""
+import os
from time import monotonic
-from ducktape.tests.test import Test
+from ducktape.utils.local_filesystem_utils import mkdir_p
+from ducktape.tests.test import Test, TestContext
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.utils.ignite_persistence import PersistenceAware, IgnitePersistenceAware
-# pylint: disable=W0223
+# pylint: disable=C0103,W0223,W0703
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,49 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_workdir(self):
+ """
+ Copy work dir from service nodes to the results directory.
+
+ If the test passed, only the default set will be collected. If the the test failed, all logs will be collected.
+ """
+ self.logger.info("copy_ignite_workdir")
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.test_context.logger.debug("Won't collect service workdir from %s." %
+ service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the service logs
+ self.test_context.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.test_context.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_work = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_work))
+ node.account.copy_from(tgz_work, dest)
+ except Exception as e:
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
done
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,45 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_root_dir(self):
+ """
+ Copying root directory from service nodes to the results directory.
+ """
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.logger.debug("Won't collect service workdir from %s." % service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the root directory
+ self.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_root = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_root))
+ node.account.copy_from(tgz_root, dest)
+ except Exception as ex:
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
technically we can just do one put, but that would be a strange test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r493829222
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,49 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_workdir(self):
+ """
+ Copy work dir from service nodes to the results directory.
+
+ If the test passed, only the default set will be collected. If the the test failed, all logs will be collected.
+ """
+ self.logger.info("copy_ignite_workdir")
+ for service in self.test_context.services:
Review comment:
you don't need to use `self.test_context.logger`, `Test` has property `logger` that do the job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494270244
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
technically we can just do one put, but that would be a strange test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494226129
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
Restart method with doc is ok. Let's do it this way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494163672
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r493407882
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ *
Review comment:
Please add meaningful java doc
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
Review comment:
Type, "ю" instead of "."
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
Am I correct that equality of dump1 and dump3 is achieved by the fact that streamer loads data the same time (1 min long for every load)?
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -136,6 +158,35 @@ def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("sudo rm -rf -- %s" % self.PERSISTENT_ROOT, allow_fail=False)
+ def rename_db(self, new_db_name: str):
+ """
+ Rename db.
+ """
+ for node in self.nodes:
+ self._rename_db(node, new_db_name)
+
+ def _rename_db(self, node, new_db_name: str):
+ """
+ Rename db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo mv {self.WORK_DIR}/db {self.WORK_DIR}/{new_db_name}", allow_fail=False)
+
+ def copy_snap_to_db(self, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ for node in self.nodes:
+ self._copy_snap_to_db(node, snapshot_name)
+
+ def _copy_snap_to_db(self, node, snapshot_name: str):
+ """
+ Copy from snapshot to db.
+ """
+ node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
+ node.account.ssh(f"sudo cp -r {self.SNAPSHOT}/{snapshot_name}/db {self.WORK_DIR}", allow_fail=False)
+ node.account.ssh(f"sudo chown -R ducker:ducker {self.PERSISTENT_ROOT}", allow_fail=False)
Review comment:
Why do you use sudo? There are chances that this operation will be forbidden on test cluster. Looks like this ownership must exist on cluster before tests run
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
Review comment:
Type, miss "." in the end of line.
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
+
+ def tearDown(self):
+ """
+ Copy work directory to result.
+ """
+ self.copy_ignite_workdir()
+
+
+def get_dump_path(control_utility: ControlUtility, node):
+ """
+ Control utils idle verify dump.
+ @return: path to dump file on node.
+ """
+ data = control_utility.idle_verify_dump(node)
+ assert 'VisorIdleVerifyDumpTask successfully' in data
+ match = re.search(r'/.*.txt', data)
+ return match[0]
+
+
+def check_validate_indexes(control_utility: ControlUtility):
+ """
+ Check indexes.
+ """
+ data = control_utility.validate_indexes()
+ assert 'no issues found.' in data, data
+
+
+def check_idle_verify(control_utility: ControlUtility):
+ """
+ Check idle verify.
+ """
+ data = control_utility.idle_verify()
+ assert 'idle_verify check has finished, no conflicts have been found.' in data, data
+
+
+def load(service_load, duration: int = 60):
Review comment:
Is it a real need to load 1 minute long?
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/UuidStreamerApplication.java
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ *
+ */
+public class UuidStreamerApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jsonNode) {
+ IgniteCache<UUID, UUID> cache = ignite.getOrCreateCache(jsonNode.get("cacheName").asText());
+
+ AtomicInteger counter = new AtomicInteger();
Review comment:
Why do you use atomic for local variable?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
Review comment:
move logging data inside `snapshot_create` (as this is the only reason for defining `data` variable) and move it to DEBUG.
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -16,12 +16,16 @@
"""
This module contains basic ignite test.
"""
+import os
from time import monotonic
-from ducktape.tests.test import Test
+from ducktape.utils.local_filesystem_utils import mkdir_p
+from ducktape.tests.test import Test, TestContext
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.utils.ignite_persistence import PersistenceAware, IgnitePersistenceAware
-# pylint: disable=W0223
+# pylint: disable=C0103,W0223,W0703
Review comment:
C0103 "Used when a name doesn't doesn't fit the naming convention associated to its type". Why do you disable it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r493826694
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
Review comment:
Typo -- should be dump_3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494169002
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,45 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_root_dir(self):
+ """
+ Copying root directory from service nodes to the results directory.
+ """
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.logger.debug("Won't collect service workdir from %s." % service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the root directory
+ self.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_root = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_root))
+ node.account.copy_from(tgz_root, dest)
+ except Exception as ex:
Review comment:
Please, inline warning W0703 and remove it from class comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 closed pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 closed pull request #8267:
URL: https://github.com/apache/ignite/pull/8267
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494264642
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,45 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_root_dir(self):
+ """
+ Copying root directory from service nodes to the results directory.
+ """
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.logger.debug("Won't collect service workdir from %s." % service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the root directory
+ self.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_root = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_root))
+ node.account.copy_from(tgz_root, dest)
+ except Exception as ex:
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8267: IGNITE-13473: Snapshot test for ducktape
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8267:
URL: https://github.com/apache/ignite/pull/8267#discussion_r494127689
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains discovery tests.
+"""
+
+import time
+import re
+
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH, IgniteVersion
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snap"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH))
+ def snapshot_test(self, ignite_version):
+ """
+ Test Snapshot.
+ 1. Start of ignite cluster.
+ 2. Activate cluster.
+ 3. Load.
+ 4. Idle verify dump1.
+ 5. Snapshot.
+ 6. Load.
+ 7. Idle verify dump2.
+ 8. Сhecking that dump1 and dump2 are differentю
+ 9. Stop ignite and replace db from snapshot
+ 10. Run ignite cluster.
+ 11. Idle verify dump3.
+ 12. Checking the equality of dump1 and dump3.
+ """
+ data_cfg = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_cfg,
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=self.NUM_NODES - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = ignite_config._replace(client_mode=True, data_storage=None)
+
+ streamer = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.UuidStreamerApplication",
+ params={"cacheName": "test-cache"})
+
+ load(streamer)
+
+ node = service.nodes[0]
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_1 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_1=" + dump_1)
+
+ data = control_utility.snapshot_create(self.SNAPSHOT_NAME)
+
+ self.logger.warn(data)
+
+ load(streamer)
+
+ dump_2 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_2=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_2}', allow_fail=True)
+ assert len(diff) != 0, diff
+
+ service.rename_db(new_db_name='old_db')
+ service.copy_snap_to_db(self.SNAPSHOT_NAME)
+
+ service.run()
+
+ control_utility.activate()
+
+ check_idle_verify(control_utility)
+ check_validate_indexes(control_utility)
+ dump_3 = get_dump_path(control_utility, node)
+
+ self.logger.warn("Path to dump_3=" + dump_2)
+
+ diff = node.account.ssh_output(f'diff {dump_1} {dump_3}', allow_fail=True)
+ assert len(diff) == 0, diff
Review comment:
No, this is because snapshot is restored
##########
File path: modules/ducktests/tests/ignitetest/utils/ignite_test.py
##########
@@ -40,3 +44,45 @@ def monotonic():
so that only the difference between the results of consecutive calls is valid.
"""
return monotonic()
+
+ def copy_ignite_root_dir(self):
+ """
+ Copying root directory from service nodes to the results directory.
+ """
+ for service in self.test_context.services:
+ if not isinstance(service, IgniteService):
+ self.logger.debug("Won't collect service workdir from %s." % service.service_id)
+ continue
+
+ if service.config.data_storage and service.config.data_storage.default.persistent:
+ # Try to copy the root directory
+ self.logger.debug("Copying persistence dir...")
+ try:
+ for node in service.nodes:
+ dest = os.path.join(
+ TestContext.results_dir(self.test_context, self.test_context.test_index),
+ service.service_id, node.account.hostname)
+ self.logger.debug("Dest dir " + dest)
+ if not os.path.isdir(dest):
+ mkdir_p(dest)
+
+ tgz_root = '%s.tgz' % PersistenceAware.PERSISTENT_ROOT
+
+ node.account.ssh(compress_cmd(PersistenceAware.PERSISTENT_ROOT, tgz_root))
+ node.account.copy_from(tgz_root, dest)
+ except Exception as ex:
Review comment:
Please, inline warning W0703 and remove it from class comment.
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -46,13 +46,40 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, modules=None):
# pylint: disable=W0221
def start(self, timeout_sec=180):
+ self._rotate_log()
+
super().start()
self.logger.info("Waiting for Ignite(s) to start...")
for node in self.nodes:
self.await_node_started(node, timeout_sec)
+ def run(self, timeout_sec=180):
Review comment:
Unfortunatelly, ducktape allows start node only with cleaning. We should add smth like restart and comment that this should be rewrittten after appropriate change in ducktape codebase will be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org