You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@geode.apache.org by GitBox <gi...@apache.org> on 2021/04/08 22:55:03 UTC
[GitHub] [geode] nonbinaryprogrammer opened a new pull request #6296: GEODE-9136: make RedisData implement Sizeable
nonbinaryprogrammer opened a new pull request #6296:
URL: https://github.com/apache/geode/pull/6296
- make RedisData implement Sizeable
- reimplement delta force recalculate size
- add tests for hash add/remove/edit wo size change
- add tests for string add/append/remove/edit wo size change
- add tests for set add/remove/edit wo size change
- add tests to verify our math is right
- keep track of the data size as part of RedisHash, RedisString,
and RedisSet
Signed-off-by: Ray Ingles <ri...@vmware.com>
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
- [ ] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
- [ ] Is your initial contribution a single, squashed commit?
- [ ] Does `gradlew build` run cleanly?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
### Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612698426
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
We can probably clean up the name some. The idea is that the operations in the "_afterDeltaPropagation()" tests don't add memory, and thus shouldn't change the bytesInUse. Not sure checking after each step is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615060908
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,514 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
Review comment:
In conjunction with the thread on the dev list, this variable could be made `final`.
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,514 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
I commented on this previously, but I think there is value for all these `AgreeUponUsedHashMemory` tests in checking at least once (maybe at the end?) that `server1` and `server2` have the same bytes in use.... especially if we want to actually test that the `membersAgreeUponUsedHashMemory`.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Could we derive the overhead via the `ReflectionObjectSizer` here and compare that to the constant instead of hard coding what we derived? Right now, someone could alter the code in a way that increases the overhead without changing the constant, but these tests wouldn't catch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-834822202
> very minor, but if you end up making other changes: if
> HASH_MAP_VALUE_PAIR_OVERHEAD & SIZE_OF_OVERHEAD_OF_FIRST_PAIR are the same value, do we need both?
take a look at `RedisHashTest.constantFirstPairOverhead_shouldEqual_calculatedOverhead`; that's where the calculation for `SIZE_OF_OVERHEAD_OF_FIRST_PAIR` happens. The value is based off `HASH_MAP_VALUE_PAIR_OVERHEAD`, but is not necessarily equal, and has different meaning. so I do believe that the two are needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r618506131
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,514 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
Unfortunately, we're having real trouble guaranteeing that things are stored in deserialized form on the secondary...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627588837
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
Review comment:
yeah I thought I had changed everything from snake case but I must have missed at least one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] upthewaterspout commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615113728
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Are we looking at hash_bytesInUse_shouldReflectActualSizeOfDataInRegion and those tests? Maybe it would be better if those tests were actually computing the size of RedisHash, rather than HashMap<ByteArrayWrapper,ByteArrayWrapper>?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615078249
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Could we derive the overhead via the `ReflectionObjectSizer` here and compare that to the constant instead of hard coding what we derived? Right now, someone could alter the code in a way that increases the overhead and forget to change the constant, but these tests wouldn't catch it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628551372
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
Review comment:
you're right. the other place we did this adds the member before doing that check, hence the `==1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] lgtm-com[bot] commented on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-827246193
This pull request **introduces 1 alert** when merging c84b2ec76118313948c7707e1ac3cc3acd2ca661 into 2e2c8f5ccbb11414e41ff03f4c7cb6e9ed38e77e - [view on LGTM.com](https://lgtm.com/projects/g/apache/geode/rev/pr-d4708faa7cf62a838be81f11154582123517e29c)
**new alerts:**
* 1 for Implicit narrowing conversion in compound assignment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r626177108
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +169,65 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ private RedisSet createRedisSetOfSpecifiedSize(int setSize) {
+ ArrayList<ByteArrayWrapper> arrayList = new ArrayList<>();
+ for (int i = 0; i < setSize; i++) {
+ arrayList.add(new ByteArrayWrapper(("abcdefgh" + i).getBytes()));
+ }
+ return new RedisSet(arrayList);
+ }
+
+ private RedisSet createRedisSetWithMemberOfSpecifiedSize(int memberSize) {
+ ArrayList<ByteArrayWrapper> arrayList = new ArrayList<>();
+ ByteArrayWrapper member =
+ new ByteArrayWrapper(createMemberOfSpecifiedSize("a", memberSize).getBytes());
+ if (member.length() > 0) {
+ arrayList.add(member);
+ }
+ return new RedisSet(arrayList);
+ }
+
+ private String createMemberOfSpecifiedSize(final String base, final int stringSize) {
+ Random random = new Random();
+ if (base.length() > stringSize) {
+ return "";
+ }
+ StringBuffer sb = new StringBuffer(stringSize);
+ sb.append(base);
+ for (int i = base.length(); i < stringSize; i++) {
+ int randy = random.nextInt(10);
+ sb.append(randy);
+ }
+ String javaString = sb.toString();
+ return javaString;
+ }
+
Review comment:
yes, good call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r613227871
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
Review comment:
That definitely makes sense. Looking through the other tests, I can see why we don't always need this deserialization trigger. The confusing part to me is that we randomly decided to use jedis2 and check server2 in this test. It is also very similar to this test: `should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease()`, but I understand the need for both.... not sure if there's a better way of naming them so people can more easily distinguish between them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627584482
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,11 +261,380 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ /************* Hash Size *************/
+ /******* constants *******/
+ @Test
+ public void constantBaseRedisHashOverhead_shouldEqualCalculatedOverhead() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisHashOverhead = reflectionObjectSizer.sizeof(new RedisHash());
+
+ assertThat(baseRedisHashOverhead).isEqualTo(BASE_REDIS_HASH_OVERHEAD);
+ }
Review comment:
they test slightly different things but since they have a lot of overlapping code, I put both assertions in this test and deleted the other one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615142156
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Is it possible to more directly unit test the calculation of those constants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r626017639
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
Review comment:
nope, we're still running up against deserialization issues. can't figure out how to force the data to be deserialized on both members, so they don't always agree on the bytes in use. these were abandoned here to be picked up in a future story (which I will go create now) because of the size of this PR and the difficulty of getting those tests running. The same is the case for the test that you commented on below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-818917838
I would feel more comfortable if @nabarunnag, @upthewaterspout and/or @dschneider-pivotal reviewed the PR as well with their knowledge of Sizeable and Geode core in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612700847
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
Review comment:
Added an intermediate step to both tests to confirm the bytesInUse actually goes up when a set/hash is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r613228101
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
I think there is value in checking at least once (maybe at the end?) that `server1` and `server2` have the same bytes in use.... especially if we want to actually test that the `membersAgreeUponUsedHashMemory` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612704490
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
Good to know. Thank you! Do you think it would be helpful to add a comment about how they were derived? (if we do, we could also add a similar comment to the `RedisSet` and `RedisString` classes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628391257
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,95 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ /************* Size in Bytes Tests *************/
+ /******* constructors *******/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
+ String javaString = makeStringOfSpecifiedSize(10_000);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int actual = string.getSizeInBytes();
+ int expected = reflectionObjectSizer.sizeof(string);
+
+ assertThat(actual).isEqualTo(expected);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofStringOfVariousSizes() {
+ String javaString;
+ for (int i = 0; i < 512; i += 8) {
+ javaString = makeStringOfSpecifiedSize(i);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int expected = reflectionObjectSizer.sizeof(string);
+ int actual = string.getSizeInBytes();
+
+ assertThat(actual).isEqualTo(expected);
+ }
+ }
+
+ /******* changing values *******/
+ @Test
+ public void changingStringValue_toShorterString_shouldDecreaseSizeInBytes() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ int initialSize = string.getSizeInBytes();
+
+ string.set(new ByteArrayWrapper(baseString.getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(initialSize - 4);
+ }
+
+ @Test
+ public void changingStringValue_toLongerString_shouldIncreaseSizeInBytes() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper(baseString.getBytes()));
+
+ int initialSize = string.getSizeInBytes();
+
+ string.set(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(initialSize + 4);
+ }
+
+ @Test
+ public void changingStringValue_toEmptyString_shouldDecreaseSizeInBytes_toBaseSize() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ string.set(new ByteArrayWrapper("".getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(BASE_REDIS_STRING_OVERHEAD);
+ }
+
+ /******* constants *******/
+ @Test
+ public void overheadConstants_shouldMatchCalculatedValue() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8); // see todo in RedisString
Review comment:
per our synchronous conversation, I have removed this assertion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-829636583
> This pull request **introduces 1 alert** when merging [c84b2ec](https://github.com/apache/geode/commit/c84b2ec76118313948c7707e1ac3cc3acd2ca661) into [2e2c8f5](https://github.com/apache/geode/commit/2e2c8f5ccbb11414e41ff03f4c7cb6e9ed38e77e) - [view on LGTM.com](https://lgtm.com/projects/g/apache/geode/rev/pr-d4708faa7cf62a838be81f11154582123517e29c)
>
> **new alerts:**
>
> * 1 for Implicit narrowing conversion in compound assignment
this has been resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612690069
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
These values were derived by empirically comparing them to the measurements from the ReflectionObjectSizer. The ReflectionObjectSizer gives an exact figure, but takes far too long (so we removed it from the final code). We confirmed that these values converge to the ROS values as the number of keys of various types increases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628595826
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
Review comment:
Thinking about it more, we don't seem to need this check at all. Similar to `membersRemoveAll`, because this method is only ever called from `applyDelta`, we actually don't need to check if the set is empty. The set will never be empty in this method. When the key/value are initially created in the region, there is no delta applied/sent (from the perspective of the `RedisSet` class).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615142156
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Could we do more direct unit testing in the `RedisHashTest.java`, maybe something like:
```
ArrayList<ByteArrayWrapper> elements = new ArrayList<>();
String field = "field";
String value = "value";
elements.add(createByteArrayWrapper(field));
elements.add(createByteArrayWrapper(value));
assertThat(ros.sizeof(new RedisHash())).isEqualTo(RedisHash.getPerHashOverhead());
assertThat(ros.sizeof(new RedisHash(elements)))
.isEqualTo(RedisHash.getPerHashOverhead() + (RedisHash.getPerStringOverhead() * 2) + field.length() + value.length());
```
...and something similar in `RedisStringTest.java` and `RedisSetTest.java`
...it should be noted that the above assertions currently fail though, so maybe that's not the correct way to test it. How did we originally derive the `PER_HASH_OVERHEAD` and `PER_STRING_OVERHEAD`? @upthewaterspout what is the proper way to derive and test those constants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628499173
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
- sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.removeAll(removes);
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ members.removeAll(removes);
Review comment:
This method does not remove every member from the set. members.removeAll() will just remove the elements contained in `removes`, so the set may not be empty afterwards. With that in mind, it makes sense to keep the code that was here previously:
```
sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
```
Because this method is only ever called from `applyDelta`, we actually don't need to check if the set is empty. The set will never be empty in this method because once the set is empty (on a different server), the key is removed from the region altogether and no `RedisSet` delta is applied/sent [see the `storeChanges` method in `RedisSet`] (so this method would not be called in that scenario).
Maybe we need to improve the way we test this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627586981
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
Review comment:
I also have no idea why it is 18. I didn't do the original Set implementation. figuring it out will take a lot of testing so I'd rather not have to. but I'll check with Ray on if he remembers where that came from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628494247
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
Review comment:
I think the condition here should be:
` if (members.isEmpty())` instead of `if (members.size() == 1)`
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
- sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.removeAll(removes);
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ members.removeAll(removes);
Review comment:
This method does not remove every member from the set. members.removeAll() will just remove the elements contained in `removes`, so the set may not be empty afterwards. With that in mind, it makes sense to keep the code that was here previously:
```
sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
```
then after the removeAll operation, check to see if members is empty. If so, set it back to the BASE_REDIS_SET_OVERHEAD. Since `membersRemoveAll` is only used in the `applyDelta` method, another option could be checking if `removes.size()` is equal to `members.size()` before looping through and subtracting the size of each member. If so, we could set it back to `BASE_REDIS_SET_OVERHEAD` and return at that point.... though I'm not sure if this will ever happen since the key is removed from the region altogether if the set is empty and no delta is set.
Maybe we need to improve the way we test this?
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
+ jedis1.sadd(SET_KEY, "other"); // two sadds are required to force
+ jedis1.sadd(SET_KEY, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(2);
+
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedStringMemory_afterDeltaPropagation() {
+ jedis1.set(STRING_KEY, "12345"); // two sets are required to force
+ jedis1.set(STRING_KEY, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.append(STRING_KEY, "a");
+ }
+
+ assertThat(jedis1.exists(STRING_KEY)).isTrue();
+ assertThat(jedis2.exists(STRING_KEY)).isTrue();
+
+ assertThat(jedis1.get(STRING_KEY)).isEqualTo("valueaaaaaaaaaa");
+ assertThat(jedis2.get(STRING_KEY)).isEqualTo("valueaaaaaaaaaa");
+
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+
+ assertThat(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
Review comment:
This will currently never be true. We could change the for loop to just do just set the key to `value` repeatedly. then make sure the STRING_KEY is still equal to `value`. Then do the same assertion we did for sets and hashes:
```
assertThat(server1FinalDataStoreBytesInUse)
.isEqualTo(server2FinalDataStoreBytesInUse)
.isEqualTo(initialDataStoreBytesInUse);
```
Unfortunately, it still doesn't pass though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] upthewaterspout commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612810877
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
+ private AtomicInteger hashSize = new AtomicInteger(PER_HASH_OVERHEAD);
Review comment:
Does this need to be an `AtomicInteger` or will just an `int` do? I see all of our update methods are synchronized, and I don't think the HashMap is safe under concurrent modification.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -145,13 +148,15 @@ private void shutDownHscanSnapshotScheduledRemoval() {
public synchronized void toData(DataOutput out, SerializationContext context) throws IOException {
super.toData(out, context);
DataSerializer.writeHashMap(hash, out);
+ DataSerializer.writeInteger(hashSize.get(), out);
Review comment:
I wonder if we actually want to serialize this field or just recompute it during deserialization?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
I wonder if we need unit tests of these constants, so we remember to update them if someone changes the code?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -46,16 +47,23 @@
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
+ private static final int PER_MEMBER_OVERHEAD = PER_OBJECT_OVERHEAD + 70;
+ private static final int PER_SET_OVERHEAD = PER_OBJECT_OVERHEAD + 240;
+
+ private AtomicInteger setSize = new AtomicInteger(PER_SET_OVERHEAD);
Review comment:
Some atomic question here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r613603512
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
done! see RedisHashTest, RedisStringTest, and RedisSetTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r618470892
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
It's been surprisingly hard to guarantee that deserialization happens for Strings and Sets. The tests end up being flaky. I've ignored them for now... if there's a way to reliably make sure both servers have deserialized versions, I'd love to hear it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615142156
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Could we do more direct unit testing in the `RedisHashTest.java`, maybe something like:
```
assertThat(ros.sizeof(new RedisHash())).isEqualTo(RedisHash.getPerHashOverhead());
assertThat(ros.sizeof(new RedisHash(createListOfDataElements(1))))
.isEqualTo(RedisHash.getPerHashOverhead() + (RedisHash.getPerStringOverhead() * 2));
```
...and something similar in `RedisStringTest.java` and `RedisSetTest.java`
...it should be noted that the above assertions currently fail though, so maybe that's not the correct way to test it. How did we originally derive the `PER_HASH_OVERHEAD` and `PER_STRING_OVERHEAD`? @upthewaterspout what is the proper way to derive and test those constants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r613603829
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
Review comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628551741
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -46,16 +46,37 @@
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
- @SuppressWarnings("unchecked")
+ // the following constants were calculated using reflection and math. you can find the tests for
+ // these values in RedisSetTest, which show the way these numbers were calculated. the constants
+ // have the advantage of saving us a lot of computation that would happen every time a new key was
+ // added. if our internal implementation changes, these values may be incorrect. the tests will
+ // catch this change. an increase in overhead should be carefully considered.
+ // Note: the per member overhead is known to not be constant. it changes as more members are
+ // added, and/or as the members get longer
+ protected static final int BASE_REDIS_SET_OVERHEAD = 112;
+ protected static final int PER_MEMBER_OVERHEAD = 77;
+ protected static final int INTERNAL_HASH_SET_STORAGE_OVERHEAD = 86;
+
+ private int sizeInBytes;
+
RedisSet(Collection<ByteArrayWrapper> members) {
+ sizeInBytes += BASE_REDIS_SET_OVERHEAD;
Review comment:
Should we move this to the empty `RedisSet` constructor like we did for `RedisHash`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r617664346
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
This turns out to be hard. We can create a ReflectionObjectSizer in the constructor, and get the size of the object without any members added.
Then we'd have to examine at least one member being added, and attempt to derive the per-object overhead as we added it. Except that it turns out that an empty HashSet doesn't allocate some storage. When you add the first object, you add both the object and the internal storage. So we could then add a second object, and based on the length attempt to figure out what the 'internal storage overhead' is...
Anyway, it's not trivial. But we might have to do it because it looks like - no surprise and as feared - JVM v11 doesn't have the same overhead as v8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r625806486
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -59,6 +59,11 @@
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private int myCalculatedSize;
Review comment:
Maybe we could call this `sizeInBytes`?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -59,6 +59,11 @@
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private int myCalculatedSize;
+ private static final int BASE_REDIS_HASH_OVERHEAD = 232;
+ protected static final int HASH_MAP_VALUE_PAIR_OVERHEAD = 96;
+ protected static final int SIZE_OF_OVERHEAD_OF_FIRST_PAIR = 96;
Review comment:
I think adding a comment here about how these were calculated and why they are constants is useful.
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
+ jedis1.sadd(SET_KEY, "other"); // two sadds are required to force
+ jedis1.sadd(SET_KEY, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(2);
+
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedStringMemory_afterDeltaPropagation() {
Review comment:
Are we able to remove this `@Ignore` now?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -35,27 +35,61 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.geode.DataSerializer;
import org.apache.geode.annotations.VisibleForTesting;
import org.apache.geode.cache.Region;
-import org.apache.geode.internal.InternalDataSerializer;
import org.apache.geode.internal.serialization.DeserializationContext;
import org.apache.geode.internal.serialization.KnownVersion;
import org.apache.geode.internal.serialization.SerializationContext;
+import org.apache.geode.internal.size.ReflectionObjectSizer;
import org.apache.geode.redis.internal.delta.AddsDeltaInfo;
import org.apache.geode.redis.internal.delta.DeltaInfo;
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
+ private static int baseRedissetOverhead;
Review comment:
If we end up keeping this variable, set should be capitalized.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -35,27 +35,61 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.geode.DataSerializer;
import org.apache.geode.annotations.VisibleForTesting;
import org.apache.geode.cache.Region;
-import org.apache.geode.internal.InternalDataSerializer;
import org.apache.geode.internal.serialization.DeserializationContext;
import org.apache.geode.internal.serialization.KnownVersion;
import org.apache.geode.internal.serialization.SerializationContext;
+import org.apache.geode.internal.size.ReflectionObjectSizer;
import org.apache.geode.redis.internal.delta.AddsDeltaInfo;
import org.apache.geode.redis.internal.delta.DeltaInfo;
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
+ private static int baseRedissetOverhead;
+ private static int perMemberOverhead;
+ private static int internalHashsetStorageOverhead;
+
+ private int myCalculatedSize;
Review comment:
Maybe we could call the variable `sizeInBytes`?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisString.java
##########
@@ -35,11 +36,15 @@
import org.apache.geode.redis.internal.netty.Coder;
public class RedisString extends AbstractRedisData {
-
private int appendSequence;
private ByteArrayWrapper value;
+ // this value is empirically derived using ReflectionObjectSizer, which provides an exact size
+ // of the object. It can't be used directly because of its performance impact. This value causes
+ // the size we keep track of to converge to the actual size as it increases.
+ public static final int PER_STRING_OVERHEAD = 74;
Review comment:
For consistency, could we give this constant a similar name to the one in `RedisHash`, maybe something like: `BASE_REDIS_STRING_OVERHEAD`?
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +169,65 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ private RedisSet createRedisSetOfSpecifiedSize(int setSize) {
+ ArrayList<ByteArrayWrapper> arrayList = new ArrayList<>();
+ for (int i = 0; i < setSize; i++) {
+ arrayList.add(new ByteArrayWrapper(("abcdefgh" + i).getBytes()));
+ }
+ return new RedisSet(arrayList);
+ }
+
+ private RedisSet createRedisSetWithMemberOfSpecifiedSize(int memberSize) {
+ ArrayList<ByteArrayWrapper> arrayList = new ArrayList<>();
+ ByteArrayWrapper member =
+ new ByteArrayWrapper(createMemberOfSpecifiedSize("a", memberSize).getBytes());
+ if (member.length() > 0) {
+ arrayList.add(member);
+ }
+ return new RedisSet(arrayList);
+ }
+
+ private String createMemberOfSpecifiedSize(final String base, final int stringSize) {
+ Random random = new Random();
+ if (base.length() > stringSize) {
+ return "";
+ }
+ StringBuffer sb = new StringBuffer(stringSize);
+ sb.append(base);
+ for (int i = base.length(); i < stringSize; i++) {
+ int randy = random.nextInt(10);
+ sb.append(randy);
+ }
+ String javaString = sb.toString();
+ return javaString;
+ }
+
Review comment:
Do we need to add tests like the ones that are in `RedisHash` (e.g. sizeShouldDecreaseWhenValueIsRemoved, shouldReturnToSetOverhead_whenAllMembersAreRemoved, etc.)?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -35,27 +35,61 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.geode.DataSerializer;
import org.apache.geode.annotations.VisibleForTesting;
import org.apache.geode.cache.Region;
-import org.apache.geode.internal.InternalDataSerializer;
import org.apache.geode.internal.serialization.DeserializationContext;
import org.apache.geode.internal.serialization.KnownVersion;
import org.apache.geode.internal.serialization.SerializationContext;
+import org.apache.geode.internal.size.ReflectionObjectSizer;
import org.apache.geode.redis.internal.delta.AddsDeltaInfo;
import org.apache.geode.redis.internal.delta.DeltaInfo;
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
+ private static int baseRedissetOverhead;
+ private static int perMemberOverhead;
+ private static int internalHashsetStorageOverhead;
+
+ private int myCalculatedSize;
+
@SuppressWarnings("unchecked")
RedisSet(Collection<ByteArrayWrapper> members) {
+ calibrate_memory_values();
+
if (members instanceof HashSet) {
this.members = (HashSet<ByteArrayWrapper>) members;
} else {
this.members = new HashSet<>(members);
}
+
+ for (ByteArrayWrapper value : this.members) {
+ myCalculatedSize += perMemberOverhead + value.length();
+ }
+ }
+
+ private void calibrate_memory_values() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ baseRedissetOverhead = reflectionObjectSizer.sizeof(this) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedissetOverhead += base_hashset_size;
+
+ ByteArrayWrapper baw1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper baw2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(baw1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ temp_hashset.add(baw2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
+ internalHashsetStorageOverhead =
+ two_entries_hashset_size - (2 * perMemberOverhead) - base_hashset_size;
+
+ myCalculatedSize = baseRedissetOverhead;
Review comment:
Since we decided to use constants for the overhead values in `RedisHash`, should we do that here as well? If not, we could move this to a static initialization block or make `sizeInBytes` static and initialize it with this static method.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -161,16 +168,46 @@ public int getDSFID() {
private synchronized ByteArrayWrapper hashPut(ByteArrayWrapper field, ByteArrayWrapper value) {
- return hash.put(field, value);
+ if (this.hash.isEmpty()) {
+ this.myCalculatedSize += SIZE_OF_OVERHEAD_OF_FIRST_PAIR;
+ }
+
+ ByteArrayWrapper oldvalue = hash.put(field, value);
+
+ if (oldvalue == null) {
+ calculateSizeOfNewFieldValuePair(field, value);
+ } else {
+ this.myCalculatedSize += value.length() - oldvalue.length();
+ }
+
+ return oldvalue;
}
private synchronized ByteArrayWrapper hashPutIfAbsent(ByteArrayWrapper field,
ByteArrayWrapper value) {
- return hash.putIfAbsent(field, value);
+ ByteArrayWrapper oldvalue = hash.putIfAbsent(field, value);
+
Review comment:
I think we need that same clause from `hashPut` here:
```
if (hash.isEmpty()) {
this.myCalculatedSize += SIZE_OF_OVERHEAD_OF_FIRST_PAIR;
}
```
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -214,19 +250,31 @@ public int getDSFID() {
}
private synchronized boolean membersAdd(ByteArrayWrapper memberToAdd) {
- return members.add(memberToAdd);
+ boolean actuallyAdded = members.add(memberToAdd);
Review comment:
Not sure if we have a convention for boolean variable names in Geode, but the general Java convention seems to be prefixing boolean variables with `is`, `has`, `can`, or `should`. With that in mind, could this variable be called `isAdded` or `isMemberAdded`?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -214,19 +250,31 @@ public int getDSFID() {
}
private synchronized boolean membersAdd(ByteArrayWrapper memberToAdd) {
- return members.add(memberToAdd);
+ boolean actuallyAdded = members.add(memberToAdd);
+ if (actuallyAdded) {
+ myCalculatedSize += perMemberOverhead + memberToAdd.length();
+ }
+ return actuallyAdded;
}
private boolean membersRemove(ByteArrayWrapper memberToRemove) {
- return members.remove(memberToRemove);
+ boolean actuallyRemoved = members.remove(memberToRemove);
Review comment:
Same comment as `actuallyAdded` above.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,11 +260,373 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ /************* Hash Size *************/
+ /******* constants *******/
+ @Test
+ public void constantValuePairOverhead_shouldEqualCalculatedOverhead() {
+ RedisHash redisHash = new RedisHash();
+ int sizeOfDataForOneFieldValuePair = 16; // initial byte[]s are 8 bytes each
+
+ HashMap<ByteArrayWrapper, ByteArrayWrapper> tempHashmap = new HashMap<>();
+
+ ByteArrayWrapper field1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper value1 = new ByteArrayWrapper("b".getBytes());
+ ByteArrayWrapper field2 = new ByteArrayWrapper("c".getBytes());
+ ByteArrayWrapper value2 = new ByteArrayWrapper("d".getBytes());
+
+ tempHashmap.put(field1, value1);
+ int oneEntryHashMapSize = reflectionObjectSizer.sizeof(tempHashmap);
+
+ tempHashmap.put(field2, value2);
+ int twoEntriesHashMapSize = reflectionObjectSizer.sizeof(tempHashmap);
+
+ int expectedValuePairOverhead = twoEntriesHashMapSize - oneEntryHashMapSize
+ - sizeOfDataForOneFieldValuePair;
+
+ assertThat(RedisHash.HASH_MAP_VALUE_PAIR_OVERHEAD).isEqualTo(expectedValuePairOverhead);
+ }
+
+ @Test
+ public void constantFirstPairOverhead_shouldEqual_calculatedOverhead() {
+ HashMap<ByteArrayWrapper, ByteArrayWrapper> tempHashmap = new HashMap<>();
+ int emptyHashMapSize = reflectionObjectSizer.sizeof(tempHashmap);
+
+ ByteArrayWrapper field = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper value = new ByteArrayWrapper("b".getBytes());
+
+ tempHashmap.put(field, value);
+ int oneEntryHashMapSize = reflectionObjectSizer.sizeof(tempHashmap);
+
+ int expectedFirstPairOverhead = oneEntryHashMapSize - emptyHashMapSize
+ - RedisHash.HASH_MAP_VALUE_PAIR_OVERHEAD;
+
+ assertThat(RedisHash.SIZE_OF_OVERHEAD_OF_FIRST_PAIR).isEqualTo(expectedFirstPairOverhead);
+ }
+
+ /******* constructor *******/
+ @Test
+ public void should_calculateSize_equalToROSSize_ofEmptyIndividualInstance() {
+ RedisHash redisHash = new RedisHash();
+
+ int expected = reflectionObjectSizer.sizeof(redisHash);
+ int actual = redisHash.getSizeInBytes();
+
+ assertThat(actual).isEqualTo(expected);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_equalToROSSize_ofIndividualInstanceWithSingleValue() {
Review comment:
Could `equalToROSSize` be changed to `closeToROSSize` like the test below it since we aren't expecting exact equality?
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -161,16 +168,46 @@ public int getDSFID() {
private synchronized ByteArrayWrapper hashPut(ByteArrayWrapper field, ByteArrayWrapper value) {
- return hash.put(field, value);
+ if (this.hash.isEmpty()) {
+ this.myCalculatedSize += SIZE_OF_OVERHEAD_OF_FIRST_PAIR;
+ }
+
+ ByteArrayWrapper oldvalue = hash.put(field, value);
+
+ if (oldvalue == null) {
+ calculateSizeOfNewFieldValuePair(field, value);
+ } else {
+ this.myCalculatedSize += value.length() - oldvalue.length();
+ }
+
+ return oldvalue;
}
private synchronized ByteArrayWrapper hashPutIfAbsent(ByteArrayWrapper field,
ByteArrayWrapper value) {
- return hash.putIfAbsent(field, value);
+ ByteArrayWrapper oldvalue = hash.putIfAbsent(field, value);
+
+ if (oldvalue == null) {
+ calculateSizeOfNewFieldValuePair(field, value);
+ }
+ return oldvalue;
+ }
+
+ private void calculateSizeOfNewFieldValuePair(ByteArrayWrapper field, ByteArrayWrapper value) {
+ this.myCalculatedSize += HASH_MAP_VALUE_PAIR_OVERHEAD + field.length() + value.length();
}
private synchronized ByteArrayWrapper hashRemove(ByteArrayWrapper field) {
- return hash.remove(field);
+ ByteArrayWrapper oldValue = hash.remove(field);
+ if (oldValue != null) {
+ myCalculatedSize -= HASH_MAP_VALUE_PAIR_OVERHEAD + field.length() + oldValue.length();
+ }
+
+ if (hash.size() == 0) {
Review comment:
We used `hash.isEmpty()` above. I realize it's basically the same, but thought we could use one or the other for consistency.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -161,16 +168,46 @@ public int getDSFID() {
private synchronized ByteArrayWrapper hashPut(ByteArrayWrapper field, ByteArrayWrapper value) {
- return hash.put(field, value);
+ if (this.hash.isEmpty()) {
+ this.myCalculatedSize += SIZE_OF_OVERHEAD_OF_FIRST_PAIR;
+ }
+
+ ByteArrayWrapper oldvalue = hash.put(field, value);
+
+ if (oldvalue == null) {
+ calculateSizeOfNewFieldValuePair(field, value);
+ } else {
+ this.myCalculatedSize += value.length() - oldvalue.length();
+ }
+
+ return oldvalue;
}
private synchronized ByteArrayWrapper hashPutIfAbsent(ByteArrayWrapper field,
ByteArrayWrapper value) {
- return hash.putIfAbsent(field, value);
+ ByteArrayWrapper oldvalue = hash.putIfAbsent(field, value);
+
+ if (oldvalue == null) {
+ calculateSizeOfNewFieldValuePair(field, value);
+ }
+ return oldvalue;
+ }
+
+ private void calculateSizeOfNewFieldValuePair(ByteArrayWrapper field, ByteArrayWrapper value) {
+ this.myCalculatedSize += HASH_MAP_VALUE_PAIR_OVERHEAD + field.length() + value.length();
Review comment:
Based on just the method name, I expected `calculateSizeOfNewFieldValuePair` to return the size of the new field/value pair. I was surprised to find it added the size directly. Maybe we could call it something like `addSizeOfNewFieldValuePair` or something to indicate it directly adds it?
Alternatively, we could just return the size of the new field value pair from this method. Then it could also be used in `hashRemove`.
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
Review comment:
Are we able to remove this `@Ignore` now?
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,50 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @Test
+ public void overheadConstants_shouldNotChange_withoutForethoughtAndTesting() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisString.getPerStringOverhead()).isEqualTo(RedisString.PER_STRING_OVERHEAD);
Review comment:
The `getPerStringOverhead` method just returns the same constant we are checking against. Could we test this constant using the ROS size of `RedisString` like we did for `RedisHash`? Also, could we change the name of the test to something like `constantRedisStringOverhead_shouldEqual_calculatedOverhead` ?
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,50 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @Test
+ public void overheadConstants_shouldNotChange_withoutForethoughtAndTesting() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisString.getPerStringOverhead()).isEqualTo(RedisString.PER_STRING_OVERHEAD);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofIndividualInstances() {
+ String javaString;
+ for (int i = 0; i < 512; i += 8) {
+ javaString = makeStringOfSpecifiedSize(i);
+ RedisString string = new RedisString(new RedisKey(javaString.getBytes()));
+
+ int expected = reflectionObjectSizer.sizeof(string);
+ Long actual = Long.valueOf(string.getSizeInBytes());
Review comment:
Intellij complains of `unnecessary boxing` for this as well as the similar line in the test below it... looks like it can just be cast to a `long` vs. using `Long.valueOf()`. Apparently this is preferred due to space/time performance.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -35,27 +35,61 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.geode.DataSerializer;
import org.apache.geode.annotations.VisibleForTesting;
import org.apache.geode.cache.Region;
-import org.apache.geode.internal.InternalDataSerializer;
import org.apache.geode.internal.serialization.DeserializationContext;
import org.apache.geode.internal.serialization.KnownVersion;
import org.apache.geode.internal.serialization.SerializationContext;
+import org.apache.geode.internal.size.ReflectionObjectSizer;
import org.apache.geode.redis.internal.delta.AddsDeltaInfo;
import org.apache.geode.redis.internal.delta.DeltaInfo;
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
+ private static int baseRedissetOverhead;
+ private static int perMemberOverhead;
+ private static int internalHashsetStorageOverhead;
Review comment:
If we end up keeping this variable, set should be capitalized.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,11 +260,373 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ /************* Hash Size *************/
+ /******* constants *******/
+ @Test
+ public void constantValuePairOverhead_shouldEqualCalculatedOverhead() {
+ RedisHash redisHash = new RedisHash();
Review comment:
This variable is unused.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,50 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @Test
+ public void overheadConstants_shouldNotChange_withoutForethoughtAndTesting() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisString.getPerStringOverhead()).isEqualTo(RedisString.PER_STRING_OVERHEAD);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofIndividualInstances() {
+ String javaString;
+ for (int i = 0; i < 512; i += 8) {
+ javaString = makeStringOfSpecifiedSize(i);
+ RedisString string = new RedisString(new RedisKey(javaString.getBytes()));
+
+ int expected = reflectionObjectSizer.sizeof(string);
+ Long actual = Long.valueOf(string.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.05));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
+ String javaString = makeStringOfSpecifiedSize(10_000);
+
+ RedisKey byteArrayWrapper = new RedisKey(javaString.getBytes());
+ RedisString string = new RedisString(byteArrayWrapper);
+
+ Long actual = Long.valueOf(string.getSizeInBytes());
+ int expected = reflectionObjectSizer.sizeof(string);
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.05));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+
+ private String makeStringOfSpecifiedSize(final int stringSize) {
+ StringBuffer sb = new StringBuffer(stringSize);
+ for (int i = 0; i < stringSize; i++) {
+ sb.append("a");
+ }
+ String javaString = sb.toString();
+ return javaString;
+ }
Review comment:
Do we need to add tests like the ones that are in `RedisHash` (e.g. sizeShouldChangeWhenValueIsChanged, shouldReturnToStringOverhead_whenValueIsRemoved, etc.)?
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,50 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @Test
+ public void overheadConstants_shouldNotChange_withoutForethoughtAndTesting() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisString.getPerStringOverhead()).isEqualTo(RedisString.PER_STRING_OVERHEAD);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofIndividualInstances() {
+ String javaString;
+ for (int i = 0; i < 512; i += 8) {
+ javaString = makeStringOfSpecifiedSize(i);
+ RedisString string = new RedisString(new RedisKey(javaString.getBytes()));
+
+ int expected = reflectionObjectSizer.sizeof(string);
+ Long actual = Long.valueOf(string.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.05));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
+ String javaString = makeStringOfSpecifiedSize(10_000);
+
+ RedisKey byteArrayWrapper = new RedisKey(javaString.getBytes());
Review comment:
It seems strange that we make a new `RedisKey` here as well as the test above this vs. creating a new `ByteArrayWrapper`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] DonalEvans merged pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
DonalEvans merged pull request #6296:
URL: https://github.com/apache/geode/pull/6296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612701188
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
Review comment:
Yup, fixing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612748271
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
Review comment:
That definitely makes sense. Looking through the other tests, I can see why we don't always need this deserialization trigger. The confusing part to me is that we randomly decided to use `jedis2` and check `server2` in this test. It is also very similar to this test: `should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease()`, but I understand the need for both.... not sure if there's a better way of naming them so people can more easily distinguish between them.
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
I think there is value in checking at least once (maybe at the end?) that `server1` and `server2` have the same bytes in use.... especially if we want to actually test that the `membersAgreeUponUsedHashMemory` .
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
Review comment:
This test is the same as `should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize`
(The only difference is that this one uses `LONG_APPEND_VALUE` instead of `value`, but that shouldn't matter since `hsetnx` does a `putifabsent` for the field)
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,526 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
Review comment:
Do we want the same checks in the `should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted()` test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-833072321
sorry, had to force push to since I rebased with develop to fix the memory overhead test that was failing because I decreased the memory overhead for hashes. I will definitely be squashing all of this out, and writing a detailed commit message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r618468958
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
I was able to get the Sets to be dynamic - however, the values didn't change for Java 8, Java 11, and Liberica v8. Hashes are more complicated - the numbers just don't seem to add up, trying to figure things out from inside the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r626952293
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
Review comment:
the jira ticket for these tests: https://issues.apache.org/jira/browse/GEODE-9243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628535583
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
+ jedis1.sadd(SET_KEY, "other"); // two sadds are required to force
+ jedis1.sadd(SET_KEY, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(2);
+
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedStringMemory_afterDeltaPropagation() {
+ jedis1.set(STRING_KEY, "12345"); // two sets are required to force
+ jedis1.set(STRING_KEY, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.append(STRING_KEY, "a");
+ }
+
+ assertThat(jedis1.exists(STRING_KEY)).isTrue();
+ assertThat(jedis2.exists(STRING_KEY)).isTrue();
+
+ assertThat(jedis1.get(STRING_KEY)).isEqualTo("valueaaaaaaaaaa");
+ assertThat(jedis2.get(STRING_KEY)).isEqualTo("valueaaaaaaaaaa");
+
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+
+ assertThat(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
Review comment:
This will currently never be true. We could change the for loop to just set the key to `value` repeatedly, then make sure the STRING_KEY is still equal to `value`. Then do the same assertion we did for sets and hashes:
```
assertThat(server1FinalDataStoreBytesInUse)
.isEqualTo(server2FinalDataStoreBytesInUse)
.isEqualTo(initialDataStoreBytesInUse);
```
Unfortunately, it still doesn't pass though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] DonalEvans commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r629632274
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -214,23 +239,43 @@ public int getDSFID() {
}
private synchronized boolean membersAdd(ByteArrayWrapper memberToAdd) {
- return members.add(memberToAdd);
+ boolean isAdded = members.add(memberToAdd);
+ if (isAdded) {
+ sizeInBytes += PER_MEMBER_OVERHEAD + memberToAdd.length();
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ }
+ return isAdded;
}
private boolean membersRemove(ByteArrayWrapper memberToRemove) {
- return members.remove(memberToRemove);
+ boolean isRemoved = members.remove(memberToRemove);
+ if (isRemoved) {
+ sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
+ }
+ return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
- return members.addAll(addsDeltaInfo.getAdds());
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
+ sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
- return members.removeAll(remsDeltaInfo.getRemoves());
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
+ if (members.size() == removes.size()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ } else {
+ sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
+ }
+ members.removeAll(removes);
}
Review comment:
I've added a test that checks this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612515511
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
Review comment:
Not a big deal, but this should be "Increase" instead of "Increases"
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
Review comment:
Are we supposed to be checking both servers here? Or checking a different server from the one our client is connected to? (Looks like we're using `jedis2`, which is connected to `server2`, but only checking the bytes for `server2`)
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
Review comment:
If we're checking that the members agree upon used memory, should we be checking the bytes in use for both server1 and server2 and asserting they are equal after each step (after we do the initial hsets and after the hsets in the for loop)?
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
Review comment:
Not a big deal, but this should be "Increase" instead of "Increases"
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
Review comment:
For this test as well as the `givenXDeleted` tests below, we aren't actually testing that the bytes decreased as the test name suggests. We are testing that they return to their initial values. The above test (`should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted`) does test that the bytes decrease.... we could potentially do both things (get the bytes before any operation, get the bytes after an operation, and get the bytes after a del, then verify the bytes decreased and then returned to their initial value)...
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
Review comment:
Same comment as above (regarding not actually testing what the test name suggests).
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
+ jedis1.sadd(SET_KEY, "value"); // two sadds are required to force
+ jedis1.sadd(SET_KEY, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showMembersAgreeUponUsedStringMemory_afterDeltaPropagation() {
Review comment:
same comment as above.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
Apologies if this is obvious, but how did we come up with the `46` and `116` here?
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ // confirm that the other member agrees upon size
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
+ jedis1.sadd(SET_KEY, "value"); // two sadds are required to force
Review comment:
same comment as above.
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
Review comment:
Not a big deal, but this should be "Increase" instead of "Increases"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627590260
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,11 +261,380 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ /************* Hash Size *************/
+ /******* constants *******/
+ @Test
+ public void constantBaseRedisHashOverhead_shouldEqualCalculatedOverhead() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisHashOverhead = reflectionObjectSizer.sizeof(new RedisHash());
+
+ assertThat(baseRedisHashOverhead).isEqualTo(BASE_REDIS_HASH_OVERHEAD);
+ }
Review comment:
ah ok, I see that now. If we want to keep both that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615106732
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
we went to go write tests for this and found that we'd already done it in PartitionedRegionStatsUpdateTest. it's the last set of tests, divided by `/**` comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628388692
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
Review comment:
through a lot more work on RedisSet, I was able to get rid of the +18. the mysterious +5 is still there, but we now know this has something to do with the way the ByteArrayWrapper resizes. hopefully we can get rid of that magic number once we finish the story on getting rid of ByteArrayWrappers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] DonalEvans commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r629632086
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -214,23 +239,43 @@ public int getDSFID() {
}
private synchronized boolean membersAdd(ByteArrayWrapper memberToAdd) {
- return members.add(memberToAdd);
+ boolean isAdded = members.add(memberToAdd);
+ if (isAdded) {
+ sizeInBytes += PER_MEMBER_OVERHEAD + memberToAdd.length();
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ }
+ return isAdded;
}
private boolean membersRemove(ByteArrayWrapper memberToRemove) {
- return members.remove(memberToRemove);
+ boolean isRemoved = members.remove(memberToRemove);
+ if (isRemoved) {
+ sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
+ }
+ return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
- return members.addAll(addsDeltaInfo.getAdds());
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
+ sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
- return members.removeAll(remsDeltaInfo.getRemoves());
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
+ if (members.size() == removes.size()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ } else {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r629501324
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -214,23 +239,43 @@ public int getDSFID() {
}
private synchronized boolean membersAdd(ByteArrayWrapper memberToAdd) {
- return members.add(memberToAdd);
+ boolean isAdded = members.add(memberToAdd);
+ if (isAdded) {
+ sizeInBytes += PER_MEMBER_OVERHEAD + memberToAdd.length();
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ }
+ return isAdded;
}
private boolean membersRemove(ByteArrayWrapper memberToRemove) {
- return members.remove(memberToRemove);
+ boolean isRemoved = members.remove(memberToRemove);
+ if (isRemoved) {
+ sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
+ }
+ return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
- return members.addAll(addsDeltaInfo.getAdds());
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
+ sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
- return members.removeAll(remsDeltaInfo.getRemoves());
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
+ if (members.size() == removes.size()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ } else {
Review comment:
We could remove this, but it doesn't hurt to leave it in.
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -214,23 +239,43 @@ public int getDSFID() {
}
private synchronized boolean membersAdd(ByteArrayWrapper memberToAdd) {
- return members.add(memberToAdd);
+ boolean isAdded = members.add(memberToAdd);
+ if (isAdded) {
+ sizeInBytes += PER_MEMBER_OVERHEAD + memberToAdd.length();
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ }
+ return isAdded;
}
private boolean membersRemove(ByteArrayWrapper memberToRemove) {
- return members.remove(memberToRemove);
+ boolean isRemoved = members.remove(memberToRemove);
+ if (isRemoved) {
+ sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
+ }
+ return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
- return members.addAll(addsDeltaInfo.getAdds());
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
+ sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
- return members.removeAll(remsDeltaInfo.getRemoves());
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
+ if (members.size() == removes.size()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ } else {
+ sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
+ }
+ members.removeAll(removes);
}
Review comment:
To test this properly, we could add a test similar to `should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation()`, but check the bytes in use after deleting some members from a set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628499173
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
- sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.removeAll(removes);
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ members.removeAll(removes);
Review comment:
This method does not remove every member from the set. members.removeAll() will just remove the elements contained in `removes`, so the set may not be empty afterwards. With that in mind, it makes sense to keep the code that was here previously:
```
sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
```
Because this method is only ever called from `applyDelta`, we actually don't need to check if the set is empty. The set will never be empty in this method because once the set is empty (on a different server), the key is removed from the region altogether and no delta is applied/sent (so this method would not be called in that scenario).
Maybe we need to improve the way we test this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r626169757
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -161,16 +168,46 @@ public int getDSFID() {
private synchronized ByteArrayWrapper hashPut(ByteArrayWrapper field, ByteArrayWrapper value) {
- return hash.put(field, value);
+ if (this.hash.isEmpty()) {
+ this.myCalculatedSize += SIZE_OF_OVERHEAD_OF_FIRST_PAIR;
+ }
+
+ ByteArrayWrapper oldvalue = hash.put(field, value);
+
+ if (oldvalue == null) {
+ calculateSizeOfNewFieldValuePair(field, value);
+ } else {
+ this.myCalculatedSize += value.length() - oldvalue.length();
+ }
+
+ return oldvalue;
}
private synchronized ByteArrayWrapper hashPutIfAbsent(ByteArrayWrapper field,
ByteArrayWrapper value) {
- return hash.putIfAbsent(field, value);
+ ByteArrayWrapper oldvalue = hash.putIfAbsent(field, value);
+
Review comment:
good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628499173
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
- sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.removeAll(removes);
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ members.removeAll(removes);
Review comment:
This method does not remove every member from the set. members.removeAll() will just remove the elements contained in `removes`, so the set may not be empty afterwards. With that in mind, it makes sense to keep the code that was here previously:
```
sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
```
Because this method is only ever called from `applyDelta`, we actually don't need to check if the set is empty. The set will never be empty in this method because once the set is empty (on a different server), the key is removed from the region altogether and no `RedisSet` delta is applied/sent (so this method would not be called in that scenario).
Maybe we need to improve the way we test this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612693930
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_ShowNoIncreaseInDatastoreBytesInUse_givenHashValueSizeDoesNotIncrease() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
Review comment:
We're actually _not_ checking both servers specifically. But it's a roll of the dice which server is the primary and which is the secondary. If we don't force a delta, the secondary server stores the initial put in serialized form, which has a different size from the deserialized form. Which means that 50% of the runs will fail, because they happen to connect to the secondary rather than the primary.
It's kind of hard to sum that up in variable names and/or comments. But if it's unclear to you, we'll probably need to revisit and see if we can make it clearer. The tests below that talk about "members agree" are supposed to actually check both servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615142156
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
Could we do some more direct unit testing in the `RedisHashTest.java`, maybe something like:
```
assertThat(ros.sizeof(new RedisHash())).isEqualTo(RedisHash.getPerHashOverhead());
assertThat(ros.sizeof(new RedisHash(createListOfDataElements(1))))
.isEqualTo(RedisHash.getPerHashOverhead() + (RedisHash.getPerStringOverhead() * 2));
```
...and something similar in `RedisStringTest.java` and `RedisSetTest.java`
...it should be noted that the above assertions currently fail though, so maybe that's not the correct way to test it. How did we originally derive the `PER_HASH_OVERHEAD` and `PER_STRING_OVERHEAD`? @upthewaterspout what is the proper way to derive/test those constants?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628390895
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedisSetOverhead += base_hashset_size;
+
+ assertThat(baseRedisSetOverhead).isEqualTo(BASE_REDIS_SET_OVERHEAD);
+ }
+
+ @Test
+ public void perMemberOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ ByteArrayWrapper member1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper member2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(member1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ temp_hashset.add(member2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ int perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
Review comment:
the magic 5 remains, after a lot more testing of RedisSet, and some changes to the implementation. it technically shouldn't be a constant, and instead is related to the size of the set, and of the bytearraywrappers within the set. it is possible for us to be more accurate here, but it requires a lot of cleverness. Since there's a story to remove ByteArrayWrappers, it seems like a waste of time to get the size exactly right all the time, when we can be within 5% without all that math.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r626952701
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -35,27 +35,61 @@
import org.apache.commons.lang3.tuple.ImmutablePair;
import org.apache.commons.lang3.tuple.Pair;
+import org.apache.geode.DataSerializer;
import org.apache.geode.annotations.VisibleForTesting;
import org.apache.geode.cache.Region;
-import org.apache.geode.internal.InternalDataSerializer;
import org.apache.geode.internal.serialization.DeserializationContext;
import org.apache.geode.internal.serialization.KnownVersion;
import org.apache.geode.internal.serialization.SerializationContext;
+import org.apache.geode.internal.size.ReflectionObjectSizer;
import org.apache.geode.redis.internal.delta.AddsDeltaInfo;
import org.apache.geode.redis.internal.delta.DeltaInfo;
import org.apache.geode.redis.internal.delta.RemsDeltaInfo;
public class RedisSet extends AbstractRedisData {
-
private HashSet<ByteArrayWrapper> members;
+ private static int baseRedissetOverhead;
+ private static int perMemberOverhead;
+ private static int internalHashsetStorageOverhead;
+
+ private int myCalculatedSize;
+
@SuppressWarnings("unchecked")
RedisSet(Collection<ByteArrayWrapper> members) {
+ calibrate_memory_values();
+
if (members instanceof HashSet) {
this.members = (HashSet<ByteArrayWrapper>) members;
} else {
this.members = new HashSet<>(members);
}
+
+ for (ByteArrayWrapper value : this.members) {
+ myCalculatedSize += perMemberOverhead + value.length();
+ }
+ }
+
+ private void calibrate_memory_values() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ baseRedissetOverhead = reflectionObjectSizer.sizeof(this) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedissetOverhead += base_hashset_size;
+
+ ByteArrayWrapper baw1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper baw2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(baw1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ temp_hashset.add(baw2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
+ internalHashsetStorageOverhead =
+ two_entries_hashset_size - (2 * perMemberOverhead) - base_hashset_size;
+
+ myCalculatedSize = baseRedissetOverhead;
Review comment:
switched to using constants and added tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612704490
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
Good to know. Thank you! Do you think it would be helpful to add a comment about how they were derived?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612642403
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,512 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncreases() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncreases() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
Review comment:
For this test as well as the `givenHashValueDeleted` test below, we aren't actually testing that the bytes decreased as the test name suggests. We are testing that they return to their initial values. The above test (`should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted`) does test that the bytes decrease.... we could potentially do both things (get the bytes before any operation, get the bytes after an operation, and get the bytes after a del, then verify the bytes decreased and then returned to their initial value)...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] ringles commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r618472482
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,514 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import org.assertj.core.data.Offset;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.internal.size.ReflectionObjectSizer;
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+ private static final int SET_SIZE = 1000;
+
+ private ReflectionObjectSizer ros = ReflectionObjectSizer.getInstance();
Review comment:
At this point the ReflectionObjectSizer isn't needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627576828
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
Review comment:
I'm pretty hesitant to do that since the old tracker tickets are still referenced and almost no one has access to them, so it's useless information. so I don't think we should be adding more ticket numbers to the codebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-834729967
Draft of commit message (feel free to leave comments):
GEODE-9136: Make RedisData implement Sizeable
- use Sizeable in RedisString, RedisSet, and RedisHash
- add unit tests of bytes in use for all three classes
- update memory overhead tests to reflect decrease in overhead
- calculate estimated size in unit tests and use constants in the classes
RedisData should implement Sizeable to increase the accuracy of the estimation of bytes in use. It is important that we are close to the right size in order for rebalancing to work properly, however the exact value is not important. These changes make the size for sets and hashes be within 5% of the size measured by using reflection. The accuracy changes as you add entries, and as entries get larger, because of the way that hashes, sets, and ByteArrayWrappers get resized. As we fill up the hash and set, we become more accurate in our estimation of the size. When full, the next entry added causes the structure to be resized, which means that our data now only accounts for a portion of the total size of the structure. We could have kept track of the resizes that happen in order to be even more accurate in our estimations, but doing so would add computational overhead for adding each key, and for each field at that key. It was decided that the benefits do not outweigh the cost.
RedisHash and RedisSet both use constants to account for the overheads of:
- an empty (RedisHash/RedisSet)
- an empty (HashMap/HashSet) within the (RedisHash/RedisSet)
- each member in the (hash/set)
RedisHash also has an overhead for the first value in the hash.
We decided to use constants in the RedisData classes in order to reduce the number of calculations we have to do every time we add a new key. The math used to derive those constants live in the RedisSetTest and RedisHashTest classes. The tests of the constants will fail if the internal implementation of the classes changes. If the overheads decrease, then the constants need to be updated to reflect that. If the overheads increase, think through the changes before adding to the overhead. Significant changes to RedisSet or RedisHash could potentially break the math, so be sure to check where the problem is, in case we need to change the calculations.
There is a magical +5 in RedisSetTest.perMemberOverheadConstant_shouldMatchCalculatedValue. We think it came from the resizing, and that 5 happens to work ok, even though it should be dynamically calculated. But as previously said, the benefits do not outweigh the cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r612704490
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -58,6 +59,9 @@
private ConcurrentHashMap<UUID, List<ByteArrayWrapper>> hScanSnapShots;
private ConcurrentHashMap<UUID, Long> hScanSnapShotCreationTimes;
private ScheduledExecutorService HSCANSnapshotExpirationExecutor = null;
+ private static final int PER_STRING_OVERHEAD = PER_OBJECT_OVERHEAD + 46;
+ private static final int PER_HASH_OVERHEAD = PER_OBJECT_OVERHEAD + 116;
Review comment:
Good to know. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r628499173
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -243,24 +251,28 @@ private boolean membersRemove(ByteArrayWrapper memberToRemove) {
boolean isRemoved = members.remove(memberToRemove);
if (isRemoved) {
sizeInBytes -= PER_MEMBER_OVERHEAD + memberToRemove.length();
+ if (members.isEmpty()) {
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ }
}
return isRemoved;
}
- private synchronized boolean membersAddAll(AddsDeltaInfo addsDeltaInfo) {
+ private synchronized void membersAddAll(AddsDeltaInfo addsDeltaInfo) {
ArrayList<ByteArrayWrapper> adds = addsDeltaInfo.getAdds();
sizeInBytes += adds.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.addAll(adds);
+ if (members.size() == 1) {
+ sizeInBytes += INTERNAL_HASH_SET_STORAGE_OVERHEAD;
+ }
+ members.addAll(adds);
}
- private synchronized boolean membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
+ private synchronized void membersRemoveAll(RemsDeltaInfo remsDeltaInfo) {
ArrayList<ByteArrayWrapper> removes = remsDeltaInfo.getRemoves();
- sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
- return members.removeAll(removes);
+ sizeInBytes = BASE_REDIS_SET_OVERHEAD;
+ members.removeAll(removes);
Review comment:
This method does not remove every member from the set. members.removeAll() will just remove the elements contained in `removes`, so the set may not be empty afterwards. With that in mind, it makes sense to keep the code that was here previously:
```
sizeInBytes -= removes.stream().mapToInt(a -> a.length() + PER_MEMBER_OVERHEAD).sum();
```
then after the removeAll operation, check to see if members is empty. If so, set it back to the BASE_REDIS_SET_OVERHEAD. Since `membersRemoveAll` is only used in the `applyDelta` method, another option could be checking if `removes.size()` is equal to `members.size()` before looping through and subtracting the size of each member. If so, we could set it back to `BASE_REDIS_SET_OVERHEAD` and return at that point.... though I'm not sure if this will ever happen since the key is removed from the region altogether and no delta is set if the set is empty.
Maybe we need to improve the way we test this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627600463
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,95 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ /************* Size in Bytes Tests *************/
+ /******* constructors *******/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
+ String javaString = makeStringOfSpecifiedSize(10_000);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int actual = string.getSizeInBytes();
+ int expected = reflectionObjectSizer.sizeof(string);
+
+ assertThat(actual).isEqualTo(expected);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofStringOfVariousSizes() {
+ String javaString;
+ for (int i = 0; i < 512; i += 8) {
+ javaString = makeStringOfSpecifiedSize(i);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int expected = reflectionObjectSizer.sizeof(string);
+ int actual = string.getSizeInBytes();
+
+ assertThat(actual).isEqualTo(expected);
+ }
+ }
+
+ /******* changing values *******/
+ @Test
+ public void changingStringValue_toShorterString_shouldDecreaseSizeInBytes() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ int initialSize = string.getSizeInBytes();
+
+ string.set(new ByteArrayWrapper(baseString.getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(initialSize - 4);
+ }
+
+ @Test
+ public void changingStringValue_toLongerString_shouldIncreaseSizeInBytes() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper(baseString.getBytes()));
+
+ int initialSize = string.getSizeInBytes();
+
+ string.set(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(initialSize + 4);
+ }
+
+ @Test
+ public void changingStringValue_toEmptyString_shouldDecreaseSizeInBytes_toBaseSize() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ string.set(new ByteArrayWrapper("".getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(BASE_REDIS_STRING_OVERHEAD);
+ }
+
+ /******* constants *******/
+ @Test
+ public void overheadConstants_shouldMatchCalculatedValue() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8); // see todo in RedisString
Review comment:
the TODO that I'm referring to is: `// TODO: for a 64bit jvm with small ops this is 12; for other 64bit jvms it is 16`. If you command-click on the `PER_OBJECT_OVERHEAD` constant, it's the TODO right above where that constant is defined in the Sizeable interface. We do need to keep the constant, at least for now, but if you feel that the test does not add value, I can remove that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 edited a comment on pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 edited a comment on pull request #6296:
URL: https://github.com/apache/geode/pull/6296#issuecomment-818917838
Just a few questions and suggested changes. Otherwise, it seems like a very valuable change! I would feel more comfortable if @nabarunnag, @upthewaterspout and/or @dschneider-pivotal reviewed the PR as well with their knowledge of Sizeable and Geode core in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] upthewaterspout commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r615091905
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,6 +258,13 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ @Test
+ public void hashSizeOverhead_shouldNotBeChanged_withoutForethoughtAndTesting() {
+ assertThat(RedisHash.PER_OBJECT_OVERHEAD).isEqualTo(8);
+ assertThat(RedisHash.getPerStringOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 46);
+ assertThat(RedisHash.getPerHashOverhead()).isEqualTo(RedisHash.PER_OBJECT_OVERHEAD + 116);
+ }
Review comment:
+1 to what Sarah said!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r613602859
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisHash.java
##########
@@ -145,13 +148,15 @@ private void shutDownHscanSnapshotScheduledRemoval() {
public synchronized void toData(DataOutput out, SerializationContext context) throws IOException {
super.toData(out, context);
DataSerializer.writeHashMap(hash, out);
+ DataSerializer.writeInteger(hashSize.get(), out);
Review comment:
from discussion with @ringles: it seems like writing it over the wire would save us some time recomputing the value. as the data set gets bigger, that could take us a good chunk of time. Is there a disadvantage to serializing it that we are not aware of?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] nonbinaryprogrammer commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
nonbinaryprogrammer commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627596380
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedisSetOverhead += base_hashset_size;
+
+ assertThat(baseRedisSetOverhead).isEqualTo(BASE_REDIS_SET_OVERHEAD);
+ }
+
+ @Test
+ public void perMemberOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ ByteArrayWrapper member1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper member2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(member1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ temp_hashset.add(member2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ int perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
Review comment:
I'm also not sure about that one. I pulled this math over from the RedisSortedSet and didn't write that original code. I'll check with Ray when he's back from his covid vaccine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627557883
##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisString.java
##########
@@ -711,6 +716,11 @@ protected void handleSetExpiration(SetOptions options) {
}
}
+ @VisibleForTesting
+ protected static int getBaseRedisStringOverhead() {
+ return BASE_REDIS_STRING_OVERHEAD;
+ }
+
Review comment:
This method is no longer used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] sabbey37 commented on a change in pull request #6296: GEODE-9136: make RedisData implement Sizeable
Posted by GitBox <gi...@apache.org>.
sabbey37 commented on a change in pull request #6296:
URL: https://github.com/apache/geode/pull/6296#discussion_r627521495
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
Review comment:
We already have the class variable `reflectionObjectSizer` that we could use instead of getting another instance.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisHashTest.java
##########
@@ -258,11 +261,380 @@ public void hscanSnaphots_shouldExpireAfterExpiryPeriod() {
});
}
+ /************* Hash Size *************/
+ /******* constants *******/
+ @Test
+ public void constantBaseRedisHashOverhead_shouldEqualCalculatedOverhead() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisHashOverhead = reflectionObjectSizer.sizeof(new RedisHash());
+
+ assertThat(baseRedisHashOverhead).isEqualTo(BASE_REDIS_HASH_OVERHEAD);
+ }
Review comment:
This test is the same as `should_calculateSize_equalToROSSize_ofEmptyIndividualInstance()`. Do we need both? Also, it looks like we already have the class variable `reflectionObjectSizer` that we could use instead of getting another instance.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
Review comment:
This could be done in one line so you wouldn't need the `temp_hashset` variable. Also, not sure why we switched to snake case for the variable name? Could we change it to `baseHashSetSize` for consistency?
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,95 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ /************* Size in Bytes Tests *************/
+ /******* constructors *******/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
+ String javaString = makeStringOfSpecifiedSize(10_000);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int actual = string.getSizeInBytes();
+ int expected = reflectionObjectSizer.sizeof(string);
+
+ assertThat(actual).isEqualTo(expected);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofStringOfVariousSizes() {
Review comment:
This title could be `equalToROSSize` since we expect exact equality.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedisSetOverhead += base_hashset_size;
+
+ assertThat(baseRedisSetOverhead).isEqualTo(BASE_REDIS_SET_OVERHEAD);
+ }
+
+ @Test
+ public void perMemberOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ ByteArrayWrapper member1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper member2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(member1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ temp_hashset.add(member2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ int perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
Review comment:
Not sure why we switched to snake case for these variable names? Could we change it to camel case for consistency?
A comment explaining the `+5` would be good.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
Review comment:
I'm wondering how we came up with `18` here? It is close to the size of an empty `ByteArrayWrapper` (which is 16)... not sure what the additional 2 would be... maybe padding? Either way, a comment would be good explaining it.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedisSetOverhead += base_hashset_size;
+
+ assertThat(baseRedisSetOverhead).isEqualTo(BASE_REDIS_SET_OVERHEAD);
+ }
+
+ @Test
+ public void perMemberOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ ByteArrayWrapper member1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper member2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(member1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ temp_hashset.add(member2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ int perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
+
+ assertThat(perMemberOverhead).isEqualTo(PER_MEMBER_OVERHEAD);
+ }
+
+ @Test
+ public void internalHashsetStorageOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ ByteArrayWrapper baw1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper baw2 = new ByteArrayWrapper("b".getBytes());
+
+ temp_hashset.add(baw1);
+ temp_hashset.add(baw2);
+
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ int internalHashsetStorageOverhead =
+ two_entries_hashset_size - (2 * PER_MEMBER_OVERHEAD) - base_hashset_size;
+
+ assertThat(internalHashsetStorageOverhead).isEqualTo(INTERNAL_HASH_SET_STORAGE_OVERHEAD);
Review comment:
Not sure why we switched to snake case for these variable names? Could we change it to camel case for consistency?
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedisSetOverhead += base_hashset_size;
+
+ assertThat(baseRedisSetOverhead).isEqualTo(BASE_REDIS_SET_OVERHEAD);
+ }
+
+ @Test
+ public void perMemberOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ ByteArrayWrapper member1 = new ByteArrayWrapper("a".getBytes());
+ ByteArrayWrapper member2 = new ByteArrayWrapper("b".getBytes());
+ temp_hashset.add(member1);
+ int one_entry_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ temp_hashset.add(member2);
+ int two_entries_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+
+ int perMemberOverhead = two_entries_hashset_size - one_entry_hashset_size + 5;
+
+ assertThat(perMemberOverhead).isEqualTo(PER_MEMBER_OVERHEAD);
+ }
+
+ @Test
+ public void internalHashsetStorageOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
Review comment:
We already have the class variable `reflectionObjectSizer` that we could use instead of getting another instance.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -165,4 +174,142 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberCounts() {
+ for (int i = 0; i < 1024; i += 16) {
+ RedisSet set = createRedisSetOfSpecifiedSize(i);
+
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROS_withVaryingMemberSize() {
+ for (int i = 0; i < 16; i++) {
+ RedisSet set = createRedisSetWithMemberOfSpecifiedSize(i * 64);
+ int expected = reflectionObjectSizer.sizeof(set);
+ Long actual = Long.valueOf(set.getSizeInBytes());
+ Offset<Long> offset = Offset.offset(Math.round(expected * 0.06));
+
+ assertThat(actual).isCloseTo(expected, offset);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void size_shouldDecrease_WhenValueIsRemoved() {
+ Region region = mock(Region.class);
+ final RedisData returnData = mock(RedisData.class);
+ when(region.put(Object.class, Object.class)).thenReturn(returnData);
+ final RedisKey key = new RedisKey("key".getBytes());
+ final ByteArrayWrapper value = new ByteArrayWrapper("value".getBytes());
+
+ RedisSet set = createRedisSetOfSpecifiedSize(10);
+
+ ArrayList<ByteArrayWrapper> members = new ArrayList<>();
+ members.add(value);
+ set.sadd(members, region, key);
+
+ int initialSize = set.getSizeInBytes();
+
+ set.srem(members, region, key);
+
+ int finalSize = set.getSizeInBytes();
+ int expectedSize = initialSize - value.length() - PER_MEMBER_OVERHEAD;
+
+ assertThat(finalSize).isEqualTo(expectedSize);
+ }
+
+ /******** constants *******/
+ @Test
+ public void baseOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
+ int baseRedisSetOverhead = reflectionObjectSizer.sizeof(new RedisSet()) + 18;
+
+ HashSet<ByteArrayWrapper> temp_hashset = new HashSet<>();
+ int base_hashset_size = reflectionObjectSizer.sizeof(temp_hashset);
+ baseRedisSetOverhead += base_hashset_size;
+
+ assertThat(baseRedisSetOverhead).isEqualTo(BASE_REDIS_SET_OVERHEAD);
+ }
+
+ @Test
+ public void perMemberOverheadConstant_shouldMatchCalculatedValue() {
+ ReflectionObjectSizer reflectionObjectSizer = ReflectionObjectSizer.getInstance();
Review comment:
We already have the class variable `reflectionObjectSizer` that we could use instead of getting another instance.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,95 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ /************* Size in Bytes Tests *************/
+ /******* constructors *******/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
Review comment:
This title could be `equalToROSSize` since we expect exact equality.
##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisStringTest.java
##########
@@ -351,4 +354,95 @@ public void setExpirationTimestamp_stores_delta_that_is_stable() throws IOExcept
o2.fromDelta(in);
assertThat(o2).isEqualTo(o1);
}
+
+ /************* Size in Bytes Tests *************/
+ /******* constructors *******/
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofLargeStrings() {
+ String javaString = makeStringOfSpecifiedSize(10_000);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int actual = string.getSizeInBytes();
+ int expected = reflectionObjectSizer.sizeof(string);
+
+ assertThat(actual).isEqualTo(expected);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void should_calculateSize_closeToROSSize_ofStringOfVariousSizes() {
+ String javaString;
+ for (int i = 0; i < 512; i += 8) {
+ javaString = makeStringOfSpecifiedSize(i);
+ RedisString string = new RedisString(new ByteArrayWrapper(javaString.getBytes()));
+
+ int expected = reflectionObjectSizer.sizeof(string);
+ int actual = string.getSizeInBytes();
+
+ assertThat(actual).isEqualTo(expected);
+ }
+ }
+
+ /******* changing values *******/
+ @Test
+ public void changingStringValue_toShorterString_shouldDecreaseSizeInBytes() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ int initialSize = string.getSizeInBytes();
+
+ string.set(new ByteArrayWrapper(baseString.getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(initialSize - 4);
+ }
+
+ @Test
+ public void changingStringValue_toLongerString_shouldIncreaseSizeInBytes() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper(baseString.getBytes()));
+
+ int initialSize = string.getSizeInBytes();
+
+ string.set(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(initialSize + 4);
+ }
+
+ @Test
+ public void changingStringValue_toEmptyString_shouldDecreaseSizeInBytes_toBaseSize() {
+ String baseString = "baseString";
+ RedisString string = new RedisString(new ByteArrayWrapper((baseString + "asdf").getBytes()));
+
+ string.set(new ByteArrayWrapper("".getBytes()));
+
+ int finalSize = string.getSizeInBytes();
+
+ assertThat(finalSize).isEqualTo(BASE_REDIS_STRING_OVERHEAD);
+ }
+
+ /******* constants *******/
+ @Test
+ public void overheadConstants_shouldMatchCalculatedValue() {
+ assertThat(RedisString.PER_OBJECT_OVERHEAD).isEqualTo(8); // see todo in RedisString
Review comment:
I'm a little confused about why we're checking this since we don't use the `PER_OBJECT_OVERHEAD` at all to calculate the size of `RedisString`. Is the todo mentioned here this one: `TODO add delta support` ?
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
+ jedis1.sadd(SET_KEY, "other"); // two sadds are required to force
+ jedis1.sadd(SET_KEY, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(2);
+
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedStringMemory_afterDeltaPropagation() {
Review comment:
Could we add the Jira ticket to the `@Ignore` comment? Like `@Ignore("GEODE-9243: Force deserialization on Radish members")`
##########
File path: geode-apis-compatible-with-redis/src/distributedTest/java/org/apache/geode/redis/internal/data/PartitionedRegionStatsUpdateTest.java
##########
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+ * agreements. See the NOTICE file distributed with this work for additional information regarding
+ * copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+ * or implied. See the License for the specific language governing permissions and limitations under
+ * the License.
+ *
+ */
+
+
+package org.apache.geode.redis.internal.data;
+
+import static org.apache.geode.distributed.ConfigurationProperties.MAX_WAIT_TIME_RECONNECT;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.Properties;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Ignore;
+import org.junit.Test;
+import redis.clients.jedis.Jedis;
+
+import org.apache.geode.test.awaitility.GeodeAwaitility;
+import org.apache.geode.test.dunit.rules.MemberVM;
+import org.apache.geode.test.dunit.rules.RedisClusterStartupRule;
+
+public class PartitionedRegionStatsUpdateTest {
+
+ @ClassRule
+ public static RedisClusterStartupRule clusterStartUpRule = new RedisClusterStartupRule(3);
+
+ private static MemberVM server1;
+ private static MemberVM server2;
+
+ private static Jedis jedis1;
+ private static Jedis jedis2;
+
+ private static final int JEDIS_TIMEOUT = Math.toIntExact(GeodeAwaitility.getTimeout().toMillis());
+ private static final String LOCAL_HOST = "127.0.0.1";
+ public static final String STRING_KEY = "string key";
+ public static final String SET_KEY = "set key";
+ public static final String HASH_KEY = "hash key";
+ public static final String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ public static final String FIELD = "field";
+
+ @BeforeClass
+ public static void classSetup() {
+ Properties locatorProperties = new Properties();
+ locatorProperties.setProperty(MAX_WAIT_TIME_RECONNECT, "15000");
+
+ MemberVM locator = clusterStartUpRule.startLocatorVM(0, locatorProperties);
+ int locatorPort = locator.getPort();
+
+ server1 = clusterStartUpRule.startRedisVM(1, locatorPort);
+ int redisServerPort1 = clusterStartUpRule.getRedisPort(1);
+ jedis1 = new Jedis(LOCAL_HOST, redisServerPort1, JEDIS_TIMEOUT);
+
+ server2 = clusterStartUpRule.startRedisVM(2, locatorPort);
+ int redisServerPort2 = clusterStartUpRule.getRedisPort(1);
+ jedis2 = new Jedis(LOCAL_HOST, redisServerPort2, JEDIS_TIMEOUT);
+ }
+
+ @Before
+ public void setup() {
+ jedis1.flushAll();
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenStringValueSizeIncreases() {
+ String LONG_APPEND_VALUE = String.valueOf(Integer.MAX_VALUE);
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.append(STRING_KEY, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(STRING_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenStringValueShortened() {
+ jedis1.set(STRING_KEY, "longer value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.set(STRING_KEY, "value");
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_resetMemoryUsage_givenFlushAllCommand() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(initialDataStoreBytesInUse).isEqualTo(0L);
+
+ jedis1.set(STRING_KEY, "value");
+
+ jedis1.flushAll();
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenStringValueSizeDoesNotIncrease() {
+ jedis1.set(STRING_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.set(STRING_KEY, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenSetValueSizeIncreases() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenSetValueSizeDoesNotIncrease() {
+ jedis1.sadd(SET_KEY, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.sadd(SET_KEY, "value");
+ }
+
+ assertThat(jedis1.scard(SET_KEY)).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.sadd(SET_KEY, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(SET_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenSetValueSizeDecreases() {
+ for (int i = 0; i < 10; i++) {
+ jedis1.sadd(SET_KEY, "value" + i);
+ }
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.srem(SET_KEY, "value" + i);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isLessThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHashValueSizeIncreases() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hset(HASH_KEY, FIELD + i, LONG_APPEND_VALUE);
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showDecreaseInDatastoreBytesInUse_givenHashValueDeleted() {
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long intermediateDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+ assertThat(intermediateDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+
+ jedis1.del(HASH_KEY);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetDoesNotIncreaseHashSize() {
+ jedis2.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis2.hset(HASH_KEY, FIELD, "value"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis2.hset(HASH_KEY, FIELD, "value");
+ }
+
+ assertThat(jedis2.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showIncreaseInDatastoreBytesInUse_givenHSetNXIncreasesHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD + i, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isGreaterThan(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ public void should_showNoIncreaseInDatastoreBytesInUse_givenHSetNXDoesNotIncreaseHashSize() {
+ jedis1.hset(HASH_KEY, FIELD, "value");
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ for (int i = 0; i < 1000; i++) {
+ jedis1.hsetnx(HASH_KEY, FIELD, "value");
+ }
+
+ long finalDataStoreBytesInUse = clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(finalDataStoreBytesInUse).isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ /******* confirm that the other member agrees upon size *******/
+
+ @Test
+ public void should_showMembersAgreeUponUsedHashMemory_afterDeltaPropagation() {
+ jedis1.hset(HASH_KEY, FIELD, "initialvalue"); // two hsets are required to force
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue"); // deserialization on both servers
+ // otherwise primary/secondary can disagree on size, and which server is primary varies
+
+ long initialDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+
+ for (int i = 0; i < 10; i++) {
+ jedis1.hset(HASH_KEY, FIELD, "finalvalue");
+ }
+
+ assertThat(jedis1.hgetAll(HASH_KEY).size()).isEqualTo(1);
+
+ long server2FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server2);
+ long server1FinalDataStoreBytesInUse =
+ clusterStartUpRule.getDataStoreBytesInUseForDataRegion(server1);
+
+ assertThat(server1FinalDataStoreBytesInUse)
+ .isEqualTo(server2FinalDataStoreBytesInUse)
+ .isEqualTo(initialDataStoreBytesInUse);
+ }
+
+ @Test
+ @Ignore("confirm that bucket size stats are being calculated correctly before enabling")
+ public void should_showMembersAgreeUponUsedSetMemory_afterDeltaPropagation() {
Review comment:
Could we add the Jira ticket to the `@Ignore` comment? Like `@Ignore("GEODE-9243: Force deserialization on Radish members")`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org