You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "torshie (via GitHub)" <gi...@apache.org> on 2023/08/16 03:31:46 UTC
[GitHub] [arrow] torshie opened a new issue, #37194: In pyarrow len(ListScalar()) may have performance issues.
torshie opened a new issue, #37194:
URL: https://github.com/apache/arrow/issues/37194
### Describe the enhancement requested
Getting the length of a list should be blazing fast, but it's not the case in pyarrow 12.0.1
I'm writing a data loading function, data are saved in parquet format. cProfile/snakeviz shows that about 20% of the time is spent on `<built-in method builtins.len>`. If the number of `len()` calls is reduced (by caching the results), the time spent can be reduced to about 13%.
I skimmed the source code of scalar.pxi and related, it seems `len(ListScalar())` does much more than reading a simple property of an object.
I'm currently using `pyarrow.compute.list_value_length` to get all lengths then cache the results, this workaround performs the best.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #37194: [Python] In pyarrow len(ListScalar()) may have performance issues
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #37194:
URL: https://github.com/apache/arrow/issues/37194#issuecomment-1688551571
In general, the Scalar classes (especially in Python) are not meant for high-performance work. Instead, you should use the vectorized operations provided by the compute functions (as you did with `list_value_length`).
That said, it would be good to reduce the overhead of the Scalar classes. If you want, you could try to contribute a PR for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] In pyarrow len(ListScalar()) may have performance issues [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #37194:
URL: https://github.com/apache/arrow/issues/37194#issuecomment-3921789039
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] In pyarrow len(ListScalar()) may have performance issues [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #37194:
URL: https://github.com/apache/arrow/issues/37194#issuecomment-4620250277
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] In pyarrow len(ListScalar()) may have performance issues [arrow]
Posted by "LucasG0 (via GitHub)" <gi...@apache.org>.
LucasG0 commented on issue #37194:
URL: https://github.com/apache/arrow/issues/37194#issuecomment-1869644463
I could not reproduce with the following code:
```python
import time
import pyarrow as pa
start = time.time()
v = ['foo', None] * 100_000_000
s = pa.scalar(v, type=pa.list_(pa.string()))
after_creation = time.time()
print(f"Time to create ListScalar object: {after_creation - start}")
_ = len(s)
after_len_call = time.time()
print(f"Time to len: {after_len_call - after_creation}")
_ = pa.compute.list_value_length(s)
print(f"Time to list_value_length: {time.time() - after_len_call}")
```
Outputs:
```
Time to create: 4.116701126098633
Time to len: 6.651878356933594e-05
Time to list_value_length: 0.0003292560577392578
```
Time spent in `len` call looks negligible here, are there some specific requirements needed to reproduce the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] In pyarrow len(ListScalar()) may have performance issues [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #37194: [Python] In pyarrow len(ListScalar()) may have performance issues
URL: https://github.com/apache/arrow/issues/37194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org