You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/09/12 08:32:24 UTC
[GitHub] [iceberg] Fokko opened a new pull request, #8549: Python: Fix caching of FileSystem
Fokko opened a new pull request, #8549:
URL: https://github.com/apache/iceberg/pull/8549
Before:
```
python3 /tmp/benchmark.py
import: 0.175605 seconds
catalog: 0.060079 seconds
table: 0.245859 seconds
scan: 5.218688 seconds
Rows: 119890
```
After:
```
python3 /tmp/benchmark.py
import: 0.169811 seconds
catalog: 0.054512 seconds
table: 0.233786 seconds
scan: 2.165105 seconds
Rows: 119890
```
https://github.com/apache/iceberg/pull/6010 didn't really fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #8549: Python: Fix caching of FileSystem
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #8549:
URL: https://github.com/apache/iceberg/pull/8549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on a diff in pull request #8549: Python: Fix caching of FileSystem
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #8549:
URL: https://github.com/apache/iceberg/pull/8549#discussion_r1322658845
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -290,7 +290,7 @@ def to_input_file(self) -> PyArrowFile:
class PyArrowFileIO(FileIO):
def __init__(self, properties: Properties = EMPTY_DICT):
- self.get_fs: Callable[[str], FileSystem] = lru_cache(self._get_fs)
+ self.get_fs: Callable[[str], FileSystem] = lru_cache(self._initialize_fs)
Review Comment:
I'm almost inclined to remove the `lru_cache` in general and make a `fs` property. We could use a single FileIO to connect to both S3 and GCS, but I'm not sure if that happens in practice, or if we should allow it. A FileIO is bound to a table, so this would mean that a table is distributed across different object stores 🤔
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -290,7 +290,7 @@ def to_input_file(self) -> PyArrowFile:
class PyArrowFileIO(FileIO):
def __init__(self, properties: Properties = EMPTY_DICT):
- self.get_fs: Callable[[str], FileSystem] = lru_cache(self._get_fs)
+ self.get_fs: Callable[[str], FileSystem] = lru_cache(self._initialize_fs)
Review Comment:
I'm almost inclined to remove the `lru_cache` in general and make a `fs` property. We could use a single FileIO to connect to both S3 and GCS, but I'm not sure if that happens in practice, or if we should allow it. A FileIO is bound to a table, so this would mean that a table is distributed across different object stores 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org