You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Zheaoli (via GitHub)" <gi...@apache.org> on 2023/04/08 13:03:33 UTC
[GitHub] [incubator-opendal] Zheaoli opened a new pull request, #1882: binding(python): Support python binding benchmark for opendal
Zheaoli opened a new pull request, #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882
In this PR, I add some benchmark code for Python binding
1. I use Gevent and greeify to patch the origin S3 client to make sure that the code run in coroutine
2. Use Opendal AsyncOperator
Here's the results
```text
Origin S3 Client async write: 0.3480771449394524
Origin S3 Client async read: 0.17831809900235385
OpenDAL S3 Client async write: 1.4063675369834527
OpenDAL S3 Client async read: 0.8196881089825183
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#issuecomment-1500913568
Weired, I tested locally and the result it:
```shell
OpenDAL S3 Client async write: 0.2892657390000295
OpenDAL S3 Client async read: 0.01604538800006594
Origin S3 Client async write: 0.21678958000006787
Origin S3 Client async read: 0.03333764000012707
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] suyanhanx commented on pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#issuecomment-1500911877
You could use `docker run -it --rm -v $(pwd):/github/workspace ghcr.io/korandoru/hawkeye-native check` to check and use `format` to fix the header issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] bxb100 commented on a diff in pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "bxb100 (via GitHub)" <gi...@apache.org>.
bxb100 commented on code in PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#discussion_r1161123323
##########
bindings/python/benchmark/requirements.txt:
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http:#www.apache.org/licenses/LICENSE-2.0
Review Comment:
# http://www.apache.org/licenses/LICENSE-2.0
pass CI
##########
bindings/python/benchmark/requirements.txt:
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http:#www.apache.org/licenses/LICENSE-2.0
Review Comment:
`# http://www.apache.org/licenses/LICENSE-2.0`
pass CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo merged pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Zheaoli commented on a diff in pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#discussion_r1161127241
##########
bindings/python/benchmark/async_origin_s3_benchmark_with_gevent.py:
##########
@@ -43,6 +44,7 @@ class Config(BaseSettings):
S3_CLIENT: S3Client = boto3_client(
"s3",
region_name=SETTINGS.aws_region,
+ endpoint_url=SETTINGS.aws_endpoint,
Review Comment:
I think this code is useless? @Xuanwo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] suyanhanx commented on pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "suyanhanx (via GitHub)" <gi...@apache.org>.
suyanhanx commented on PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#issuecomment-1500913540
How did you run the benchmark? With local minio?
How about using this framework https://pyperformance.readthedocs.io/ ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#issuecomment-1500916422
After some simple change:
```
OpenDAL S3 Client async write: 0.07601107799996498
OpenDAL S3 Client async read: 0.015181756999936624
Origin S3 Client async write: 0.2072812530000192
Origin S3 Client async read: 0.03448412199986706
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] bxb100 commented on a diff in pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "bxb100 (via GitHub)" <gi...@apache.org>.
bxb100 commented on code in PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#discussion_r1161123468
##########
bindings/python/benchmark/requirements.txt:
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http:#www.apache.org/licenses/LICENSE-2.0
Review Comment:
`# http://www.apache.org/licenses/LICENSE-2.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on a diff in pull request #1882: binding(python): Support python binding benchmark for opendal
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #1882:
URL: https://github.com/apache/incubator-opendal/pull/1882#discussion_r1161127484
##########
bindings/python/benchmark/async_origin_s3_benchmark_with_gevent.py:
##########
@@ -43,6 +44,7 @@ class Config(BaseSettings):
S3_CLIENT: S3Client = boto3_client(
"s3",
region_name=SETTINGS.aws_region,
+ endpoint_url=SETTINGS.aws_endpoint,
Review Comment:
> I think this code is useless? @Xuanwo
If not set, we will visit AWS S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org