You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "sebpretzer (via GitHub)" <gi...@apache.org> on 2023/12/18 18:34:36 UTC
[PR] moto server implementation for pytest [iceberg-python]
sebpretzer opened a new pull request, #223:
URL: https://github.com/apache/iceberg-python/pull/223
This is the PR relating to https://github.com/apache/iceberg-python/issues/222.
This should be a drop in replacement for `moto.mock_s3`, by pointing the s3 clients at a moto server endpoint url.
By doing this, you can:
1. Remove the patch to `aiobotocore`
2. Remove the requirement of using `pyiceberg.io.fsspec.FsspecFileIO` over the default of `pyiceberg.io.pyarrow.PyArrowFileIO`.
Glue and DynamoDB still use the basic `mock_glue` and `mock_dynamodb`. They could probably be moved over to the sever implementation as well, but it was taking more work than I wanted so I didn't include it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Moto server implementation for pytest [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #223:
URL: https://github.com/apache/iceberg-python/pull/223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] moto server implementation for pytest [iceberg-python]
Posted by "sebpretzer (via GitHub)" <gi...@apache.org>.
sebpretzer commented on code in PR #223:
URL: https://github.com/apache/iceberg-python/pull/223#discussion_r1430842215
##########
tests/conftest.py:
##########
@@ -1661,11 +1586,33 @@ def fixture_aws_credentials() -> Generator[None, None, None]:
os.environ.pop("AWS_DEFAULT_REGION")
+MOTO_SERVER = ThreadedMotoServer(port=5456)
Review Comment:
Fixed! I re-ran the tests on my end.
Aside: I actually tracked down what was [actually happening](https://nono.ma/port-5000-used-by-control-center-in-macos-ventura-controlce), it was not something I had stood up at 5000... 🤦♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] moto server implementation for pytest [iceberg-python]
Posted by "sebpretzer (via GitHub)" <gi...@apache.org>.
sebpretzer commented on code in PR #223:
URL: https://github.com/apache/iceberg-python/pull/223#discussion_r1430837527
##########
tests/conftest.py:
##########
@@ -1661,11 +1586,33 @@ def fixture_aws_credentials() -> Generator[None, None, None]:
os.environ.pop("AWS_DEFAULT_REGION")
+MOTO_SERVER = ThreadedMotoServer(port=5456)
Review Comment:
No particular reason. The [default is 5000](https://github.com/getmoto/moto/blob/a8d7dfd7af198838152687dd80867653e6e5126a/moto/moto_server/threaded_moto_server.py#L12). I think I had something running at 5000 when I had initially set this up. Let me drop it in favor of the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] moto server implementation for pytest [iceberg-python]
Posted by "sebpretzer (via GitHub)" <gi...@apache.org>.
sebpretzer commented on code in PR #223:
URL: https://github.com/apache/iceberg-python/pull/223#discussion_r1430842215
##########
tests/conftest.py:
##########
@@ -1661,11 +1586,33 @@ def fixture_aws_credentials() -> Generator[None, None, None]:
os.environ.pop("AWS_DEFAULT_REGION")
+MOTO_SERVER = ThreadedMotoServer(port=5456)
Review Comment:
Fixed!
Aside: I actually tracked down what was [actually happening](https://nono.ma/port-5000-used-by-control-center-in-macos-ventura-controlce), it was not something I had stood up at 5000... 🤦♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] moto server implementation for pytest [iceberg-python]
Posted by "HonahX (via GitHub)" <gi...@apache.org>.
HonahX commented on code in PR #223:
URL: https://github.com/apache/iceberg-python/pull/223#discussion_r1430830405
##########
tests/conftest.py:
##########
@@ -1661,11 +1586,33 @@ def fixture_aws_credentials() -> Generator[None, None, None]:
os.environ.pop("AWS_DEFAULT_REGION")
+MOTO_SERVER = ThreadedMotoServer(port=5456)
Review Comment:
Just out of curiosity, why do we go with port '5456'? Is there a particular reason or was it more of a random pick?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] moto server implementation for pytest [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #223:
URL: https://github.com/apache/iceberg-python/pull/223#issuecomment-1861517385
cc @HonahX since he's the original author of the Glue catalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org