You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Fokko (via GitHub)" <gi...@apache.org> on 2023/01/26 16:25:17 UTC
[GitHub] [iceberg] Fokko opened a new pull request, #6673: Python: Optimize PyArrow reads 🚀🚀🚀
Fokko opened a new pull request, #6673:
URL: https://github.com/apache/iceberg/pull/6673
PyArrow is still sluggish when it comes into opening files, and we still see many requests being made to S3.
This PR removes the Dataset, and uses the lower read_table API. Since the read_table API requires to pass in filters in the DNF form, we need to do some additional conversion.
This PR reduces the number of calls from 203 to 165. Requests log:
Before: https://gist.github.com/Fokko/96b4d5b65ec85c95d6e875f6ec19bf50
After: https://gist.github.com/Fokko/282f6b803d83a830465d97f64cf10057
Query used:
```python
from pyiceberg.catalog import load_catalog
catalog = load_catalog('local')
tbl = catalog.load_table('nyc.taxis')
from pyiceberg.expressions import GreaterThanOrEqual, LessThanOrEqual, And
sc = tbl.scan(row_filter=And(
GreaterThanOrEqual("tpep_pickup_datetime", "2022-04-01T00:00:00.000000+00:00"),
LessThanOrEqual("tpep_pickup_datetime", "2022-04-28T00:00:00.000000+00:00"),
)).to_arrow()
```
Logs:
Also, the wall clock time is lower:
```python
➜ iceberg git:(fd-optimize-pyarrow) ✗ time python3 /tmp/vo.py
python3 /tmp/vo.py 2.38s user 2.75s system 31% cpu 16.067 total
python3 /tmp/vo.py 2.55s user 2.57s system 36% cpu 14.097 total
python3 /tmp/vo.py 2.60s user 2.57s system 32% cpu 15.954 total
```
```python
➜ iceberg git:(master) time python3 /tmp/vo.py
python3 /tmp/vo.py 2.54s user 2.71s system 28% cpu 18.499 total
python3 /tmp/vo.py 2.75s user 2.56s system 24% cpu 21.547 total
python3 /tmp/vo.py 2.75s user 2.95s system 17% cpu 32.554 total
```
Keep in mind that these requests are across the great ocean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6673: Python: Optimize PyArrow reads
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #6673:
URL: https://github.com/apache/iceberg/pull/6673#discussion_r1088468333
##########
python/pyiceberg/io/pyarrow.py:
##########
@@ -493,31 +495,32 @@ def _file_to_table(
)
file_schema = Schema.parse_raw(schema_raw)
- pyarrow_filter = None
- if bound_row_filter is not AlwaysTrue():
- translated_row_filter = translate_column_names(bound_row_filter, file_schema, case_sensitive=case_sensitive)
- bound_file_filter = bind(file_schema, translated_row_filter, case_sensitive=case_sensitive)
- pyarrow_filter = expression_to_pyarrow(bound_file_filter)
-
- file_project_schema = prune_columns(file_schema, projected_field_ids, select_full_types=False)
-
- if file_schema is None:
- raise ValueError(f"Missing Iceberg schema in Metadata for file: {path}")
-
- # Prune the stuff that we don't need anyway
- file_project_schema_arrow = schema_to_pyarrow(file_project_schema)
-
- read_options = {
- "pre_buffer": True,
- "use_buffered_stream": True,
- "buffer_size": 8388608,
- }
+ pyarrow_filter = None
+ dnf_filter = None
+ if bound_row_filter is not AlwaysTrue():
+ translated_row_filter = translate_column_names(bound_row_filter, file_schema, case_sensitive=case_sensitive)
+ bound_file_filter = bind(file_schema, translated_row_filter, case_sensitive=case_sensitive)
+ dnf_filter = to_plain_format(rewrite_to_dnf(bound_file_filter), cast_int_to_datetime=True)
+ pyarrow_filter = expression_to_pyarrow(bound_file_filter)
+
+ file_project_schema = prune_columns(file_schema, projected_field_ids, select_full_types=False)
+
+ if file_schema is None:
+ raise ValueError(f"Missing Iceberg schema in Metadata for file: {path}")
+
+ arrow_table = pq.read_table(
+ source=fout,
Review Comment:
:rocket:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #6673: Python: Optimize PyArrow reads
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #6673:
URL: https://github.com/apache/iceberg/pull/6673#discussion_r1092389857
##########
python/pyiceberg/expressions/visitors.py:
##########
@@ -940,7 +963,9 @@ def visit_or(
raise ValueError(f"Not allowed: {left_result} || {right_result}")
-def expression_to_plain_format(expressions: Tuple[BooleanExpression, ...]) -> List[List[Tuple[str, str, Any]]]:
+def expression_to_plain_format(
+ expressions: Tuple[BooleanExpression, ...], cast_int_to_datetime: bool = False
Review Comment:
When would this not be set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #6673: Python: Optimize PyArrow reads
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #6673:
URL: https://github.com/apache/iceberg/pull/6673#issuecomment-1410953886
Thanks, @Fokko! Nice work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #6673: Python: Optimize PyArrow reads
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #6673:
URL: https://github.com/apache/iceberg/pull/6673#issuecomment-1405849535
Looks good to me when tests are passing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Fokko commented on pull request #6673: Python: Optimize PyArrow reads
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #6673:
URL: https://github.com/apache/iceberg/pull/6673#issuecomment-1406227312
@rdblue thanks for the review. This one is blocked by https://github.com/apache/iceberg/pull/6566
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #6673: Python: Optimize PyArrow reads
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #6673:
URL: https://github.com/apache/iceberg/pull/6673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org