You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/12/14 18:51:40 UTC
[GitHub] [ignite] Sega76 opened a new pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Sega76 opened a new pull request #8575:
URL: https://github.com/apache/ignite/pull/8575
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549803615
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
index_validate and idle_verify take a long time
`01:03:33 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
01:03:33 status: PASS
01:03:33 run time: 4 minutes 56.490 seconds
01:03:33 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
01:03:33 status: PASS
01:03:33 run time: 23 minutes 57.322 seconds`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549641738
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
1) do we really should check the condition "(0 < start_time < end_time)" here?
2) should we have an assert here "(snapshot_name == name)" instead of check since LastSnapshotName MUST show the last started snapshot name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549700429
##########
File path: modules/ducktests/tests/ignitetest/services/utils/jmx_utils.py
##########
@@ -76,7 +76,7 @@ def find_mbean(self, pattern, domain='org.apache'):
:param domain: Domain of MBean
:return: JmxMBean instance
"""
- cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '%s'" \
+ cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '.*%s$'" \
Review comment:
we can pass a short bean name(ZookeeperDiscoverySpi) instead of
* group = SPIs, name = ZookeeperDiscoverySpi
##########
File path: modules/ducktests/tests/ignitetest/services/utils/jmx_utils.py
##########
@@ -76,7 +76,7 @@ def find_mbean(self, pattern, domain='org.apache'):
:param domain: Domain of MBean
:return: JmxMBean instance
"""
- cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '%s'" \
+ cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '.*%s$'" \
Review comment:
I found it more convenient. undo the change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562015433
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
Review comment:
Don't see tests where it runs repeatedly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r555154735
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550141947
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
Review comment:
we may run it repeatedly, key should be different
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r555153692
##########
File path: modules/ducktests/tests/ignitetest/services/utils/jmx_utils.py
##########
@@ -76,7 +76,7 @@ def find_mbean(self, pattern, domain='org.apache'):
:param domain: Domain of MBean
:return: JmxMBean instance
"""
- cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '%s'" \
+ cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '.*%s$'" \
Review comment:
returned
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562471869
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
LastSnapshotName displays the name of the last snapshot on a node from all nodes, if we go to the wrong node, we can get information about the previous snapshot, then the snapshot name may be different
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561974889
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
1. Suggest answering the question.
2. That's the exact situation I'm talking about. Another snapshot in progress means we doing something wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562478670
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
example:
node.account.hostname == ducker02
node.account.externally_routable_ip == ducker02
we need to use an ip address: 172.22.0.3
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561957476
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
Review comment:
On, now we have "amount". Amount of what?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550162635
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
Review comment:
mbean.LastSnapshotEndTime is python generator -> str
we may get next value with next()
then it will look like this
int(next(mbean.LastSnapshotEndTime))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r543281753
##########
File path: modules/ducktests/tests/ignitetest/services/utils/ignite_aware.py
##########
@@ -351,3 +351,25 @@ def __dump_netfilter_settings(node):
Reads current netfilter settings on the node for debugging purposes.
"""
return str(node.account.ssh_client.exec_command("sudo iptables -L -n")[1].read(), sys.getdefaultencoding())
+
+ def restart(self):
+ """
+ Restart ignite cluster without cleaning.
+ """
+ self.stop()
+ self.rotate_log()
+ self.start(clean=False)
+
+ def rotate_log(self):
+ """
+ Rotate log file.
+ """
+ new_log_file = self.STDOUT_STDERR_CAPTURE.replace('.log', '_$N.log')
+
+ for node in self.nodes:
+ node.account.ssh(f'if [ -e {self.STDOUT_STDERR_CAPTURE} ];'
+ f'then '
+ f'cd {self.LOGS_DIR};'
+ f'N=`ls | grep log | wc -l`;'
Review comment:
when starting different services on the same nodes (for example, when testing the PDS), we will not be able to use counter the service.
So I think the counter is a bad idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562483011
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
+ """
+ :return: path to database directory
+ """
+ return os.path.join(self.work_dir, "db")
+
+ @property
+ def snapshots_dir(self):
+ """
+ :return: path to snapshots directory
+ """
+ return os.path.join(self.work_dir, "snapshots")
+
def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("rm -rf -- %s" % self.persistent_root, allow_fail=False)
+ def rename_database(self, new_name: str):
Review comment:
removed this method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r573708997
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, DEV_BRANCH, LATEST_2_9
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=4)
+ @ignite_versions(str(DEV_BRANCH), str(LATEST_2_9))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=DataStorageConfiguration(default=DataRegionConfiguration(persistent=True)),
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi'
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=len(self.test_context.cluster) - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = IgniteConfiguration(
+ client_mode=True,
+ version=IgniteVersion(ignite_version),
+ discovery_spi=from_ignite_cluster(service)
+ )
+
+ loader = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.snapshot_test.DataLoaderApplication",
+ shutdown_timeout_sec=300,
+ params={
+ "cacheName": self.CACHE_NAME,
+ "interval": 500_000,
+ "dataSizeKB": 1
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549700898
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
Review comment:
we can run this test by passing a parameter through globals
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561957476
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
Review comment:
On, now we have "amount". Amount of what?
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
Line 87 is empty.
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
Let me ask again,
what is the reason to have dedicated "initialized" and "finished" states?
Why not only the markSyncExecutionComplete() used?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
Review comment:
"Node where will be dump file"
Could you please specify an action in this sentence?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
Review comment:
Could you please answer my questions?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
1. Suggest answering the question.
2. That's the exact situation I'm talking about. Another snapshot in progress means we doing something wrong.
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
+
+ raise TimeoutError(f'LastSnapshotName={name}, '
+ f'LastSnapshotStartTime={start_time}, '
+ f'LastSnapshotEndTime={end_time}, '
+ f'LastSnapshotErrorMessage={err_msg}')
Review comment:
Did not understand the answer.
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
>> node.account.externally_routable_ip is hostname
so, what the difference between "hostname" (node.account.externally_routable_ip) and node.account.hostname?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
> node.account.externally_routable_ip is hostname
so, what the difference between "hostname" (node.account.externally_routable_ip) and node.account.hostname?
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
Do we really need so long checks?
Why it takes so long? Do we check snapshots properly?
Why Dev is so slow?
@Mmuzaf Could you please check this?
Seems we have Dev 5 times slower than 2.9.0
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
Review comment:
Why LATEST_2_9?
Will it be latest after the 2.10 release?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
Review comment:
any reason to have a dedicated variable?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
Review comment:
Could you please answer the question I asked?
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
Review comment:
Don't see tests where it runs repeatedly.
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
+ """
+ :return: path to database directory
+ """
+ return os.path.join(self.work_dir, "db")
+
+ @property
+ def snapshots_dir(self):
+ """
+ :return: path to snapshots directory
+ """
+ return os.path.join(self.work_dir, "snapshots")
+
def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("rm -rf -- %s" % self.persistent_root, allow_fail=False)
+ def rename_database(self, new_name: str):
Review comment:
Seems you changed `mv` to `cp`, what the reason to perform a so heavy operation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r563751121
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562008682
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
Review comment:
any reason to have a dedicated variable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
`================================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
`==============================================================================02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
02:06:30 ================================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
02:06:30 --------------------------------------------------------------------------------
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562586010
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
long execution due to checking the indexes:
2021-01-22 14:41:10,996 - remoteaccount - _log - lineno:160]: ducker@tkles-pprb00277.vm.esrt.cloud.sbrf.ru: Running ssh command: /opt/ignite-dev/bin/control.sh --host 10.53.211.13 --cache validate_indexes
2021-01-22 14:53:49,009 - control_utility - __run - lineno:331]: Output of command --cache validate_indexes on node tkles-pprb00277.vm.esrt.cloud.sbrf.ru, exited with code 0, is Control utility [ver. 2.10.0-SNAPSHOT#20210122-sha1:DEV]
2021 Copyright(C) Apache Software Foundation
User: ducker
Time: 2021-01-22T14:41:12.376
Command [CACHE] started
Arguments: --host 10.53.211.13 --cache validate_indexes
--------------------------------------------------------------------------------
no issues found.
Command [CACHE] finished with code: 0
Control utility has completed execution at: 2021-01-22T14:53:48.986
Execution time: 756610 ms
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549635931
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
Review comment:
success execution assert missed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] timoninmaxim commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r543254570
##########
File path: modules/ducktests/tests/ignitetest/services/utils/ignite_aware.py
##########
@@ -351,3 +351,25 @@ def __dump_netfilter_settings(node):
Reads current netfilter settings on the node for debugging purposes.
"""
return str(node.account.ssh_client.exec_command("sudo iptables -L -n")[1].read(), sys.getdefaultencoding())
+
+ def restart(self):
Review comment:
make it private
##########
File path: modules/ducktests/tests/ignitetest/services/utils/ignite_aware.py
##########
@@ -351,3 +351,25 @@ def __dump_netfilter_settings(node):
Reads current netfilter settings on the node for debugging purposes.
"""
return str(node.account.ssh_client.exec_command("sudo iptables -L -n")[1].read(), sys.getdefaultencoding())
+
+ def restart(self):
+ """
+ Restart ignite cluster without cleaning.
+ """
+ self.stop()
+ self.rotate_log()
+ self.start(clean=False)
+
+ def rotate_log(self):
+ """
+ Rotate log file.
+ """
+ new_log_file = self.STDOUT_STDERR_CAPTURE.replace('.log', '_$N.log')
+
+ for node in self.nodes:
+ node.account.ssh(f'if [ -e {self.STDOUT_STDERR_CAPTURE} ];'
+ f'then '
+ f'cd {self.LOGS_DIR};'
+ f'N=`ls | grep log | wc -l`;'
Review comment:
Use grep with full pattern patching, smth like "^console_?[0-9]*.log$". As string "log" may match different log files.
Alternative is using service counter variable that incremented within restart method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r555151063
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549805206
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
Review comment:
so as not to take up CPU time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
`
================================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562009881
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
Review comment:
Could you please answer the question I asked?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549700429
##########
File path: modules/ducktests/tests/ignitetest/services/utils/jmx_utils.py
##########
@@ -76,7 +76,7 @@ def find_mbean(self, pattern, domain='org.apache'):
:param domain: Domain of MBean
:return: JmxMBean instance
"""
- cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '%s'" \
+ cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '.*%s$'" \
Review comment:
we can pass a short bean name(ZookeeperDiscoverySpi) instead of
* group = SPIs, name = ZookeeperDiscoverySpi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561992594
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
>> node.account.externally_routable_ip is hostname
so, what the difference between "hostname" (node.account.externally_routable_ip) and node.account.hostname?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
> node.account.externally_routable_ip is hostname
so, what the difference between "hostname" (node.account.externally_routable_ip) and node.account.hostname?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549626817
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
+
+ try (IgniteDataStreamer<UUID, byte[]> dataStreamer = ignite.dataStreamer(cacheName)) {
+ for (long j = 0L; j <= size; j++)
Review comment:
1) why "j" ? :)
2) why "<=" ?
3) any reason to specify "L"?
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
what is the reason to have a dedicated "initialized" state?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
1) do we really should check the condition "(0 < start_time < end_time)" here?
2) should we have an assert here "(snapshot_name == name)" instead of check since LastSnapshotName MUST show the last tarted snapshot name?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
Review comment:
in java code it looks like `LongMetric startTime = mreg0.findMetric("LastSnapshotStartTime");`
why it looks so strange - "int(list(...)[0])" in python?
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
why this suite is slow? how about to make it fast?
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
Review comment:
Why it streams UUID as a key?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
Review comment:
params description missed
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
what is the difference with "node.account.externally_routable_ip" used before?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
Review comment:
"Node" param description missed
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
Review comment:
any real reason to have sync_mode param when it's always true in code?
any reason to replace an explicit call of await_snapshot by this param?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
Review comment:
assert missed
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
+
+ raise TimeoutError(f'LastSnapshotName={name}, '
+ f'LastSnapshotStartTime={start_time}, '
+ f'LastSnapshotEndTime={end_time}, '
+ f'LastSnapshotErrorMessage={err_msg}')
Review comment:
it seems we have all the chances to gain "name" from one snapshot, "start_time" from another, and "err_msg" from the third
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
Review comment:
what is "size" when we have "dataSize"?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
Review comment:
reason to have slept here?
##########
File path: modules/ducktests/tests/ignitetest/services/utils/jmx_utils.py
##########
@@ -76,7 +76,7 @@ def find_mbean(self, pattern, domain='org.apache'):
:param domain: Domain of MBean
:return: JmxMBean instance
"""
- cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '%s'" \
+ cmd = "echo $'open %s\\n beans -d %s \\n close' | %s | grep -o '.*%s$'" \
Review comment:
please explain the changes
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
Review comment:
should this be defined at IgnitePathAware?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
Review comment:
used only as default at @cluster
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
Review comment:
why it may take so long time to startup?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
Review comment:
Why not LAST version used?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
Review comment:
any reason to have this variable?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
Review comment:
any reason to have this variable?
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = IgniteConfiguration(
+ client_mode=True,
+ version=IgniteVersion(ignite_version),
+ discovery_spi=from_ignite_cluster(service),
+ )
+
+ loader = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.snapshot_test.UuidDataLoaderApplication",
+ params={
+ "cacheName": self.CACHE_NAME,
+ "size": 512 * 1024,
+ "dataSize": 1024
+ },
+ startup_timeout_sec=180,
+ shutdown_timeout_sec=300
+ )
+
+ loader.run()
+
+ node = service.nodes[0]
Review comment:
why it is assigned here but the next code line does not use it?
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
+ """
+ :return: path to database directory
+ """
+ return os.path.join(self.work_dir, "db")
+
+ @property
+ def snapshots_dir(self):
+ """
+ :return: path to snapshots directory
+ """
+ return os.path.join(self.work_dir, "snapshots")
+
def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("rm -rf -- %s" % self.persistent_root, allow_fail=False)
+ def rename_database(self, new_name: str):
Review comment:
it performs "mv" but has the "rename*" name...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549803615
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
index_validate and idle_verify take a long time
01:03:33 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
01:03:33 status: PASS
01:03:33 run time: 4 minutes 56.490 seconds
01:03:33 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
01:03:33 status: PASS
01:03:33 run time: 23 minutes 57.322 seconds
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561967442
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
Review comment:
Could you please answer my questions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov merged pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov merged pull request #8575:
URL: https://github.com/apache/ignite/pull/8575
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549697531
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -248,12 +311,15 @@ def __parse_cluster_state(output):
return ClusterState(state=state, topology_version=topology, baseline=baseline)
- def __run(self, cmd):
- node = random.choice(self.__alives())
+ def __run(self, cmd, node=None):
+ if node is None:
+ node = random.choice(self.__alives())
self.logger.debug(f"Run command {cmd} on node {node.name}")
- raw_output = node.account.ssh_capture(self.__form_cmd(node, cmd), allow_fail=True)
+ node_ip = socket.gethostbyname(node.account.hostname)
Review comment:
node.account.externally_routable_ip is hostname
for the dump file to be generated on a specific node, we must pass ip
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561958877
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
Line 87 is empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550175008
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561963740
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
Review comment:
"Node where will be dump file"
Could you please specify an action in this sentence?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r563753120
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
+
+ raise TimeoutError(f'LastSnapshotName={name}, '
+ f'LastSnapshotStartTime={start_time}, '
+ f'LastSnapshotEndTime={end_time}, '
+ f'LastSnapshotErrorMessage={err_msg}')
Review comment:
fixed error message
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
`
================================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------
`
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
`================================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------`
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
`==============================================================================02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------`
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
==============================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
02:06:30 ================================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
02:06:30 --------------------------------------------------------------------------------
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
===============================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
02:06:30 --------------------------------------------------------------------------------
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
02:06:30 --------------------------------------------------------------------------------
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549803615
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
index_validate and idle_verify take a long time
`01:03:33 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
01:03:33 status: PASS
01:03:33 run time: 4 minutes 56.490 seconds
01:03:33 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
01:03:33 status: PASS
01:03:33 run time: 23 minutes 57.322 seconds`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
===============================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
02:06:30 --------------------------------------------------------------------------------
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562586010
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
long execution due to checking the indexes:
[DEBUG - 2021-01-22 14:41:10,996 - remoteaccount - _log - lineno:160]: ducker@tkles-pprb00277.vm.esrt.cloud.sbrf.ru: Running ssh command: /opt/ignite-dev/bin/control.sh --host 10.53.211.13 --cache validate_indexes
[DEBUG - 2021-01-22 14:53:49,009 - control_utility - __run - lineno:331]: Output of command --cache validate_indexes on node tkles-pprb00277.vm.esrt.cloud.sbrf.ru, exited with code 0, is Control utility [ver. 2.10.0-SNAPSHOT#20210122-sha1:DEV]
2021 Copyright(C) Apache Software Foundation
User: ducker
Time: 2021-01-22T14:41:12.376
Command [CACHE] started
Arguments: --host 10.53.211.13 --cache validate_indexes
--------------------------------------------------------------------------------
no issues found.
Command [CACHE] finished with code: 0
Control utility has completed execution at: 2021-01-22T14:53:48.986
Execution time: 756610 ms
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550174049
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550174087
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r555151919
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562467982
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
if something goes wrong, during operation, then we will consider it during initialization, I think this is not correct
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] ivandasch commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
ivandasch commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r566076989
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, DEV_BRANCH, LATEST_2_9
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=4)
+ @ignite_versions(str(DEV_BRANCH), str(LATEST_2_9))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=DataStorageConfiguration(default=DataRegionConfiguration(persistent=True)),
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi'
+ )
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=len(self.test_context.cluster) - 1)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = IgniteConfiguration(
+ client_mode=True,
+ version=IgniteVersion(ignite_version),
+ discovery_spi=from_ignite_cluster(service)
+ )
+
+ loader = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.snapshot_test.DataLoaderApplication",
+ shutdown_timeout_sec=300,
+ params={
+ "cacheName": self.CACHE_NAME,
+ "interval": 500_000,
+ "dataSizeKB": 1
Review comment:
valueSizeKb is better imho
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r560871668
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549801511
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = IgniteConfiguration(
+ client_mode=True,
+ version=IgniteVersion(ignite_version),
+ discovery_spi=from_ignite_cluster(service),
+ )
+
+ loader = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.snapshot_test.UuidDataLoaderApplication",
+ params={
+ "cacheName": self.CACHE_NAME,
+ "size": 512 * 1024,
+ "dataSize": 1024
+ },
+ startup_timeout_sec=180,
+ shutdown_timeout_sec=300
+ )
+
+ loader.run()
+
+ node = service.nodes[0]
Review comment:
so that the work of the control utility was in one block, redo it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550133823
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
+ """
+ :return: path to database directory
+ """
+ return os.path.join(self.work_dir, "db")
+
+ @property
+ def snapshots_dir(self):
+ """
+ :return: path to snapshots directory
+ """
+ return os.path.join(self.work_dir, "snapshots")
+
def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("rm -rf -- %s" % self.persistent_root, allow_fail=False)
+ def rename_database(self, new_name: str):
Review comment:
"move" implies getting a new path. we rename the base folder
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562586010
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
long execution due to checking the indexes:
```
2021-01-22 14:41:10,996 - remoteaccount - _log - lineno:160]: ducker@tkles-pprb00277.vm.esrt.cloud.sbrf.ru: Running ssh command: /opt/ignite-dev/bin/control.sh --host 10.53.211.13 --cache validate_indexes
2021-01-22 14:53:49,009 - control_utility - __run - lineno:331]: Output of command --cache validate_indexes on node tkles-pprb00277.vm.esrt.cloud.sbrf.ru, exited with code 0, is Control utility [ver. 2.10.0-SNAPSHOT#20210122-sha1:DEV]
2021 Copyright(C) Apache Software Foundation
User: ducker
Time: 2021-01-22T14:41:12.376
Command [CACHE] started
Arguments: --host 10.53.211.13 --cache validate_indexes
--------------------------------------------------------------------------------
no issues found.
Command [CACHE] finished with code: 0
Control utility has completed execution at: 2021-01-22T14:53:48.986
Execution time: 756610 ms
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550162779
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
File "/opt/ignite-dev/modules/ducktests/tests/ignitetest/tests/snapshot_test.py", line 87, in snapshot_test
loader.run()
File "/usr/local/lib/python3.7/dist-packages/ducktape/services/service.py", line 322, in run
self.start()
File "/opt/ignite-dev/modules/ducktests/tests/ignitetest/services/utils/ignite_aware.py", line 85, in start
self.await_started()
File "/opt/ignite-dev/modules/ducktests/tests/ignitetest/services/ignite_app.py", line 51, in await_started
self.__check_status("IGNITE_APPLICATION_INITIALIZED", timeout=self.startup_timeout_sec)
need "initialized" state
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r563744531
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
02:06:30 --------------------------------------------------------------------------------
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562480283
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
Review comment:
I think that after the release of 2.10
it will need to be added to the test, because 2.9 should also be tested
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562484898
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
Review comment:
Startup takes the same amount of time as in other tests with PDS.
Time increased for local startup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562004702
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
Review comment:
Why LATEST_2_9?
Will it be latest after the 2.10 release?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550131032
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
Review comment:
in the following tests, will need to kill the node while taking a snapshot
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r563744531
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
Review comment:
fixed
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
+
+ raise TimeoutError(f'LastSnapshotName={name}, '
+ f'LastSnapshotStartTime={start_time}, '
+ f'LastSnapshotEndTime={end_time}, '
+ f'LastSnapshotErrorMessage={err_msg}')
Review comment:
fixed error message
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r563753470
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562001736
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
Do we really need so long checks?
Why it takes so long? Do we check snapshots properly?
Why Dev is so slow?
@Mmuzaf Could you please check this?
Seems we have Dev 5 times slower than 2.9.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549806368
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
1. you suggest removing 0 (start_time < end_time)?
2. if we take 2 snapshots in succession, we may will get an exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r570611482
##########
File path: modules/ducktests/tests/ignitetest/tests/suites/slow_suite.yml
##########
@@ -15,3 +15,6 @@
discovery:
- ../discovery_test.py
+
+snapshot:
+ - ../snapshot_test.py
Review comment:
fixed the classpath
==============================================================================
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=2.9.0
02:06:30 status: PASS
02:06:30 run time: 5 minutes 17.370 seconds
02:06:30 --------------------------------------------------------------------------------
02:06:30 test_id: ignitetest.tests.snapshot_test.SnapshotTest.snapshot_test.ignite_version=dev
02:06:30 status: PASS
02:06:30 run time: 5 minutes 23.372 seconds
--------------------------------------------------------------------------------
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r543284503
##########
File path: modules/ducktests/tests/ignitetest/services/utils/ignite_aware.py
##########
@@ -351,3 +351,25 @@ def __dump_netfilter_settings(node):
Reads current netfilter settings on the node for debugging purposes.
"""
return str(node.account.ssh_client.exec_command("sudo iptables -L -n")[1].read(), sys.getdefaultencoding())
+
+ def restart(self):
Review comment:
made a private rotate_log
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562019628
##########
File path: modules/ducktests/tests/ignitetest/services/ignite.py
##########
@@ -38,10 +38,47 @@ def __init__(self, context, config, num_nodes, jvm_opts=None, startup_timeout_se
super().__init__(context, config, num_nodes, startup_timeout_sec, shutdown_timeout_sec, modules=modules,
jvm_opts=jvm_opts)
+ @property
+ def database_dir(self):
+ """
+ :return: path to database directory
+ """
+ return os.path.join(self.work_dir, "db")
+
+ @property
+ def snapshots_dir(self):
+ """
+ :return: path to snapshots directory
+ """
+ return os.path.join(self.work_dir, "snapshots")
+
def clean_node(self, node):
node.account.kill_java_processes(self.APP_SERVICE_CLASS, clean_shutdown=False, allow_fail=True)
node.account.ssh("rm -rf -- %s" % self.persistent_root, allow_fail=False)
+ def rename_database(self, new_name: str):
Review comment:
Seems you changed `mv` to `cp`, what the reason to perform a so heavy operation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r555156262
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549804804
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
+
+ raise TimeoutError(f'LastSnapshotName={name}, '
+ f'LastSnapshotStartTime={start_time}, '
+ f'LastSnapshotEndTime={end_time}, '
+ f'LastSnapshotErrorMessage={err_msg}')
Review comment:
cannot run multiple snapshot operations
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549806368
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
Review comment:
1. you suggest removing 0 (start_time < end_time)
2. if we take 2 snapshots in succession, we may will get an exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561988292
##########
File path: modules/ducktests/tests/ignitetest/services/utils/control_utility.py
##########
@@ -127,6 +131,65 @@ def tx_kill(self, **kwargs):
res = self.__parse_tx_list(output)
return res if res else output
+ def validate_indexes(self):
+ """
+ Validate indexes.
+ """
+ data = self.__run("--cache validate_indexes")
+
+ assert ('no issues found.' in data), data
+
+ def idle_verify(self):
+ """
+ Idle verify.
+ """
+ data = self.__run("--cache idle_verify")
+
+ assert ('idle_verify check has finished, no conflicts have been found.' in data), data
+
+ def idle_verify_dump(self, node=None):
+ """
+ Idle verify dump.
+ """
+ data = self.__run("--cache idle_verify --dump", node=node)
+
+ assert ('VisorIdleVerifyDumpTask successfully' in data), data
+
+ return re.search(r'/.*.txt', data).group(0)
+
+ def snapshot_create(self, snapshot_name: str, sync_mode: bool = True, timeout_sec: int = 60):
+ """
+ Create snapshot.
+ """
+ res = self.__run(f"--snapshot create {snapshot_name}")
+
+ if ("Command [SNAPSHOT] finished with code: 0" in res) & sync_mode:
+ self.await_snapshot(snapshot_name, timeout_sec)
+
+ def await_snapshot(self, snapshot_name: str, timeout_sec=60):
+ """
+ Waiting for the snapshot to complete.
+ """
+ delta_time = datetime.now() + timedelta(seconds=timeout_sec)
+
+ while datetime.now() < delta_time:
+ for node in self._cluster.nodes:
+ mbean = JmxClient(node).find_mbean('snapshot')
+ start_time = int(list(mbean.LastSnapshotStartTime)[0])
+ end_time = int(list(mbean.LastSnapshotEndTime)[0])
+ err_msg = list(mbean.LastSnapshotErrorMessage)[0]
+ name = list(mbean.LastSnapshotName)[0]
+
+ if (snapshot_name == name) and (0 < start_time < end_time) and (err_msg == ''):
+ return
+
+ time.sleep(1)
+
+ raise TimeoutError(f'LastSnapshotName={name}, '
+ f'LastSnapshotStartTime={start_time}, '
+ f'LastSnapshotEndTime={end_time}, '
+ f'LastSnapshotErrorMessage={err_msg}')
Review comment:
Did not understand the answer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r550173934
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
+
+ try (IgniteDataStreamer<UUID, byte[]> dataStreamer = ignite.dataStreamer(cacheName)) {
+ for (long j = 0L; j <= size; j++)
Review comment:
1. set "i"
2. set "<"
3. use "int"
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r561962164
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
+ /** {@inheritDoc} */
+ @Override public void run(JsonNode jNode) {
+ String cacheName = jNode.get("cacheName").asText();
+
+ long size = jNode.get("size").asLong();
+
+ int dataSize = jNode.get("dataSize").asInt();
+
+ markInitialized();
Review comment:
Let me ask again,
what is the reason to have dedicated "initialized" and "finished" states?
Why not only the markSyncExecutionComplete() used?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r555156588
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
+ service.start()
+
+ control_utility = ControlUtility(service, self.test_context)
+ control_utility.activate()
+
+ client_config = IgniteConfiguration(
+ client_mode=True,
+ version=IgniteVersion(ignite_version),
+ discovery_spi=from_ignite_cluster(service),
+ )
+
+ loader = IgniteApplicationService(
+ self.test_context,
+ client_config,
+ java_class_name="org.apache.ignite.internal.ducktest.tests.snapshot_test.UuidDataLoaderApplication",
+ params={
+ "cacheName": self.CACHE_NAME,
+ "size": 512 * 1024,
+ "dataSize": 1024
+ },
+ startup_timeout_sec=180,
+ shutdown_timeout_sec=300
+ )
+
+ loader.run()
+
+ node = service.nodes[0]
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r549800770
##########
File path: modules/ducktests/tests/ignitetest/tests/snapshot_test.py
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Module contains snapshot test.
+"""
+from ducktape.mark.resource import cluster
+
+from ignitetest.services.ignite import IgniteService
+from ignitetest.services.ignite_app import IgniteApplicationService
+from ignitetest.services.utils.control_utility import ControlUtility
+from ignitetest.services.utils.ignite_configuration import IgniteConfiguration, DataStorageConfiguration
+from ignitetest.services.utils.ignite_configuration.cache import CacheConfiguration
+from ignitetest.services.utils.ignite_configuration.data_storage import DataRegionConfiguration
+from ignitetest.services.utils.ignite_configuration.discovery import from_ignite_cluster
+from ignitetest.utils import ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import IgniteVersion, V_2_9_0, DEV_BRANCH
+
+
+# pylint: disable=W0223
+class SnapshotTest(IgniteTest):
+ """
+ Test Snapshot.
+ """
+ NUM_NODES = 4
+
+ SNAPSHOT_NAME = "test_snapshot"
+
+ CACHE_NAME = "TEST_CACHE"
+
+ @cluster(num_nodes=NUM_NODES)
+ @ignite_versions(str(DEV_BRANCH), str(V_2_9_0))
+ def snapshot_test(self, ignite_version):
+ """
+ Basic snapshot test.
+ """
+ data_storage = DataStorageConfiguration(default=DataRegionConfiguration(persistent=True))
+
+ ignite_config = IgniteConfiguration(
+ version=IgniteVersion(ignite_version),
+ data_storage=data_storage,
+ metric_exporter='org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi',
+ caches=[CacheConfiguration(name=self.CACHE_NAME, cache_mode='REPLICATED',
+ indexed_types=['java.util.UUID', 'byte[]'])]
+ )
+
+ num_nodes = len(self.test_context.cluster)
+
+ service = IgniteService(self.test_context, ignite_config, num_nodes=num_nodes - 1, startup_timeout_sec=180)
Review comment:
for local tests like in PmeFreeSwitchTest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] Sega76 commented on a change in pull request #8575: IGNITE-13492: Added snapshot test to ducktests
Posted by GitBox <gi...@apache.org>.
Sega76 commented on a change in pull request #8575:
URL: https://github.com/apache/ignite/pull/8575#discussion_r562482610
##########
File path: modules/ducktests/src/main/java/org/apache/ignite/internal/ducktest/tests/snapshot_test/UuidDataLoaderApplication.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.ducktest.tests.snapshot_test;
+
+import java.util.UUID;
+import com.fasterxml.jackson.databind.JsonNode;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.internal.ducktest.utils.IgniteAwareApplication;
+
+/**
+ * Loading random uuids to cache.
+ */
+public class UuidDataLoaderApplication extends IgniteAwareApplication {
Review comment:
SnapshotTest
we run it repeatedly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org