You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ChenSammi (via GitHub)" <gi...@apache.org> on 2023/09/06 05:28:21 UTC
[GitHub] [ozone] ChenSammi opened a new pull request, #5250: HDDS-9240. Test container schema V2 and V3 replica mutual replication
ChenSammi opened a new pull request, #5250:
URL: https://github.com/apache/ozone/pull/5250
https://issues.apache.org/jira/browse/HDDS-9240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual import
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1726832466
@captainzmc , could you help to review the patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual import
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1727178290
Looks like the new test is flaky. Since this is a test-only change, I've reverted it.
```
Tests run: 176, Failures: 0, Errors: 16, Skipped: 6, Time elapsed: 61.031 s <<< FAILURE! - in org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer
Error: TestKeyValueContainer.testImportV3ReplicaToV2HddsVolume Time elapsed: 0.116 s <<< ERROR!
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.lambda$setUp$2(TestKeyValueContainer.java:174)
at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:40)
at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:99)
at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:33)
at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:57)
at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:147)
at org.apache.hadoop.ozone.container.common.volume.RoundRobinVolumeChoosingPolicy.chooseVolume(RoundRobinVolumeChoosingPolicy.java:50)
at org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testMixedSchemaImport(TestKeyValueContainer.java:986)
at org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testImportV3ReplicaToV2HddsVolume(TestKeyValueContainer.java:970)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual import
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1727341300
> Looks like the new test is flaky. Since this is a test-only change, I've reverted it.
>
> ```
> Tests run: 176, Failures: 0, Errors: 16, Skipped: 6, Time elapsed: 61.031 s <<< FAILURE! - in org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer
> Error: TestKeyValueContainer.testImportV3ReplicaToV2HddsVolume Time elapsed: 0.116 s <<< ERROR!
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:659)
> at java.util.ArrayList.get(ArrayList.java:435)
> at org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.lambda$setUp$2(TestKeyValueContainer.java:174)
> at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:40)
> at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:99)
> at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
> at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:33)
> at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:57)
> at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:147)
> at org.apache.hadoop.ozone.container.common.volume.RoundRobinVolumeChoosingPolicy.chooseVolume(RoundRobinVolumeChoosingPolicy.java:50)
> at org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testMixedSchemaImport(TestKeyValueContainer.java:986)
> at org.apache.hadoop.ozone.container.keyvalue.TestKeyValueContainer.testImportV3ReplicaToV2HddsVolume(TestKeyValueContainer.java:970)
> ```
The failure is because #5300 changed some code in TestKeyValueContainer#setUp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual replication
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1708003967
These are unit tests, not integration tests. If you look at the rest of the tests in ReplicationSupervisor, they have mocked the import / export parts out, and test only the features of the ReplicationSupervisor - limits, expiring old entries, metric counting etc. This new test is very different, and involves setting up volumes etc to facilitate testing the importer and exporter, which are part of the KeyValueContainer class. There should be unit level tests in KeyValueContainer which validate that units behavior, which in this case is importing V2 to V3. The tests will also likely be easier to write and understand, and probably faster to execute.
Another side issue is that there are two code paths in the replicator tasks now - the push and pull replication modes. Pull is basically dead - we don't use it anymore in the new replication manager, which is the default now. The test added here I think uses the download and import code path, which is the old model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc merged pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual import
Posted by "captainzmc (via GitHub)" <gi...@apache.org>.
captainzmc merged PR #5250:
URL: https://github.com/apache/ozone/pull/5250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual replication
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1707942620
@sodonnel , the test at the ReplicationSupervisor and DownloadAndImportReplicator can mimic the real container replication case at most extent, the only omit part is the container tar file transmission with GRPC channel. All the related functions in KeyValueContainer will be called during the test, including container creation, container export and container import. I think the current way can cover more logic in the container replication path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual import
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1727722361
@captainzmc , thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual replication
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1707915115
I haven't looked at this in detail, but I think a test like this would be better suited in TestKeyValueContainer, where there are already some other import test cases, such as empty container etc. That way, you are testing the import flow directly.
At the replication supervisor level, it just pushes a blob of data around and it doesn't really care what scheme it is - whether it is V2 or V3 is handled by the exporter and importer code which is all inside KeyValueContainer.java I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5250: HDDS-9240. Test container schema V2 and V3 replica mutual import
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5250:
URL: https://github.com/apache/ozone/pull/5250#issuecomment-1709666889
>
> Another side issue is that there are two code paths in the replicator tasks now - the push and pull replication modes. Pull is basically dead - we don't use it anymore in the new replication manager, which is the default now. The test added here I think uses the download and import code path, which is the old model.
Okay. I will move the test location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org