You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@geode.apache.org by "jujoramos (GitHub)" <gi...@apache.org> on 2020/03/25 17:01:43 UTC

[GitHub] [geode] jujoramos opened pull request #4848: GEODE-7670: Add Tests for PR clear

Added distributed tests to verify that the clear operation on
Partitioned Regions works as expected when there are other
concurrent operations happening on the cache (puts, gets, members
added and members removed).

Thank you for submitting a contribution to Apache Geode.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

- [X] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?

- [X] Is your initial contribution a single, squashed commit?

- [X] Does `gradlew build` run cleanly?

- [X] Have you written or updated unit tests to verify your changes?

- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?

### Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.


[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

This is a good start; other cache ops that can be added in mix are Region ops (destroyRegion, invalidateRegion), querying; Region with index, rebalance, etc.  

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] gesterzhou commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "gesterzhou (GitHub)" <gi...@apache.org>.

5 seconds is rvv domination timeout. If clear operation took more than 5 seconds, something must be wrong. No need to measure other operations' time. 


[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

This is going over the PRs entry/key set getting all the values. This will be consistent where-ever this is called. 
Are we trying to see if the data in one vm is same as in another vm. Or are we trying to see data in primary buckets are same as in secondary buckets.
Fetching the local PR data-set and iterating over it will give the data stored in that vm. 
Or just to see an op could be executed successfully after clear?

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

No need to specify time; if someone changes the time and doesn't update it, this will become incorrect.

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on issue #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

@gesterzhou 
I've rebased the `PR` against `feature/GEODE-7665` to include the latest changes from the `clear` feature. There are still some failures while running the tests, concurrent `putAll` with `clear` (test method `clearWithConcurrentPutAllShouldWorkCorrectly`) always reproduces a discrepancy between primary buckets a secondary buckets for `PARTITION_REDUNDANT` regions.
Other tests also sporadically fail when I uncomment the following lines in the `assertRegionBucketsConsistency` method too:
```
assertThat(otherDump.getRvv())
    .as("RegionVersionVector for bucket " + bucketId + " on member " 
        + otherDump.getMember() + " is not consistent with member " + firstDump.getMember())
    .isEqualTo(firstDump.getRvv());
```

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

3 secs long sleep.

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] gesterzhou commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "gesterzhou (GitHub)" <gi...@apache.org>.

it's better to change the continuous put into continuous putAll. 

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

Agreed, the `DistributionMessageObserver` can be used but it's a test hook and I've heard we want to get rid of that eventually, that's why I chose to use another approach (which seems to work fine, so I'm not sure I should to apply the requested change here).

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

Cool, will add another test to execute multiple `putAll` instead of regular `put` opertations.

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

Changed.

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

Same as above, I'm still inclined to use the `CacheWriter` for now instead of the `DistributionMessageObserver` hook. If the lifecycle of the listener changes, I'll be more than happy to use another approach :-). If you think this should block the PR from being merged, though, I'll explore using the `DistributionMessageObserver`.

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

The method asserts that the data is consistent across VMs by tacking a snapshot of the region per VM and comparing it against the other running VMs, it doesn't check primaries vs secondaries.

```
    vms.forEach(vm -> vm.invoke(() -> {
      final Map<String, String> thisSnapshot = waitForSilenceAndGetRegionSnapshot();
      assertThat(thisSnapshot).isEqualTo(vm0Snapshot);
      thisSnapshot.forEach((key, value) -> assertThat(value).isEqualTo("Value_" + key));
    }));

```

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

Shortening the test name :)
We have PRConcurrentMapOpsJUnitTest - similar to this?

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

I won't foresee removal of observers; again there are multiple ways to do it; it is your choice :)  

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] agingade commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "agingade (GitHub)" <gi...@apache.org>.

Instead of CacheWriter, you can use DistributionMessageObserver. And can use VM.bounce* to kill vms. No need to wait for CacheWriter messaging implementation. 

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org

[GitHub] [geode] jujoramos commented on pull request #4848: GEODE-7670: Add Tests for PR clear

Posted by "jujoramos (GitHub)" <gi...@apache.org>.

Changed.

[ Full content available at: https://github.com/apache/geode/pull/4848 ]
This message was relayed via gitbox.apache.org for notifications@geode.apache.org