You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/09/06 13:32:55 UTC
[GitHub] [iceberg] xingbowu commented on pull request #3067: Aliyun: Mock aliyun OSS(Object Storage Service)
xingbowu commented on pull request #3067:
URL: https://github.com/apache/iceberg/pull/3067#issuecomment-913654757
> Thanks @xingbowu to make a smaller PR for the Aliyun [OSS integration](https://github.com/apache/iceberg/pull/2230) work. I skimmed the whole PR, this PR is trying to introduce the aliyun oss mock app server, so that we could build oss test cases on top of it (We don't need to mock all the called OSS API in a relative complex test cases , such as multi-upload test cases). It's a good thing for us to introduce a simple simulator to align the local mock app and aliyun online oss server.
>
> The most important thing for me is: How do we guarantee the local mock app is aligning correctly to the aliyun online oss server ? In the parent PR #2230 , we introduced a class test rule named OSSTestRule and the rule has two different impl:
>
> * OSSMockRule: The rule will start a local mini aliyun oss server, which could serving the remote OSS client http requests.
> * OSSIntegrationTestRule: The rule will prepare testing buckets in the remote oss server, so that the test cases could write real data to.
>
> For the local oss application, we provide tests cases in [TestLocalOSS](https://github.com/apache/iceberg/blob/3755c967f61035097abbebfe419d17473bf5fefd/aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/TestLocalOSS.java) , which was designed to run different TestRule according to the intentional environment variables (will run local mock app by default). If the tests could be passes on both local env and remote oss env, then we could definitely ensure the local oss app is implemented correctly.
> Thanks @xingbowu to make a smaller PR for the Aliyun [OSS integration](https://github.com/apache/iceberg/pull/2230) work. I skimmed the whole PR, this PR is trying to introduce the aliyun oss mock app server, so that we could build oss test cases on top of it (We don't need to mock all the called OSS API in a relative complex test cases , such as multi-upload test cases). It's a good thing for us to introduce a simple simulator to align the local mock app and aliyun online oss server.
>
> The most important thing for me is: How do we guarantee the local mock app is aligning correctly to the aliyun online oss server ? In the parent PR #2230 , we introduced a class test rule named OSSTestRule and the rule has two different impl:
>
> * OSSMockRule: The rule will start a local mini aliyun oss server, which could serving the remote OSS client http requests.
> * OSSIntegrationTestRule: The rule will prepare testing buckets in the remote oss server, so that the test cases could write real data to.
>
> For the local oss application, we provide tests cases in [TestLocalOSS](https://github.com/apache/iceberg/blob/3755c967f61035097abbebfe419d17473bf5fefd/aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/TestLocalOSS.java) , which was designed to run different TestRule according to the intentional environment variables (will run local mock app by default). If the tests could be passes on both local env and remote oss env, then we could definitely ensure the local oss app is implemented correctly.
Thanks a lot for pointing out missing part. I have done rework and added local test here to guarantee quality. comparing with 2230, implemented more test case, such as range get.
Additionally, followed up your review comments to simulate basic aliyun oss behavior including create/put/delete and excluded multi-parts , feel free to let me know if you have further comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org