You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "aersam (via GitHub)" <gi...@apache.org> on 2024/02/19 07:49:16 UTC
[I] from_pylist should allow a parameter to scan more records for columns [arrow]
aersam opened a new issue, #40124:
URL: https://github.com/apache/arrow/issues/40124
### Describe the enhancement requested
This:
```python
import pyarrow
print(pyarrow.Table.from_pylist([{"a": 1, "b": 2}, {"a": 3, "b": 4, "c": 5}]))
```
results in
```
a: int64
b: int64
----
a: [[1,3]]
b: [[2,4]]
```
I think it's kind of okay, but also a bit surprising. I think there should at least be parameter so that the result would rather be:
```
a: int64
b: int64
c: int64
----
a: [[1,3]]
b: [[2,4]]
c: [[Null,5]]
```
There are API's that do not return properties in case they are null, if you call from_pylist for such one, you will loose data. In my case the API was [the Power BI API](https://learn.microsoft.com/en-us/rest/api/power-bi/admin/reports-get-reports-as-admin).
One can of you course pass the schema to workaround this, but that's not always a good option (eg, you'll loose additional new properties)
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] from_pylist should allow a parameter to scan more records for columns [arrow]
Posted by "aersam (via GitHub)" <gi...@apache.org>.
aersam commented on issue #40124:
URL: https://github.com/apache/arrow/issues/40124#issuecomment-1951946801
Btw, Polars does this "correctly"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] from_pylist should allow a parameter to scan more records for columns [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #40124:
URL: https://github.com/apache/arrow/issues/40124#issuecomment-1966532178
The current behaviour is somewhat documented:
schema : Schema, default None
If not passed, will be inferred from the first row of the
mapping values.
But I agree this might be surprising, or in any case that it can be useful to change that behaviour.
Workaround on the short term is to use `pa.array` to infer the list of dicts to a StructArray, which has the desired behaviour, and then this array can be converted zero copy to a batch or table:
```
In [4]: arr = pa.array([{"a": 1, "b": 2}, {"a": 3, "b": 4, "c": 5}])
In [5]: batch = pa.RecordBatch.from_struct_array(arr)
In [6]: batch
Out[6]:
pyarrow.RecordBatch
a: int64
b: int64
c: int64
----
a: [1,3]
b: [2,4]
c: [null,5]
In [7]: batch.to_pandas()
Out[7]:
a b c
0 1 2 NaN
1 3 4 5.0
```
I suppose for larger data, this should actually also be faster, and so we should maybe consider using that under the hood as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] from_pylist should allow a parameter to scan more records for columns [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #40124:
URL: https://github.com/apache/arrow/issues/40124#issuecomment-3921313856
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] from_pylist should allow a parameter to scan more records for columns [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #40124:
URL: https://github.com/apache/arrow/issues/40124#issuecomment-3547081533
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] from_pylist should allow a parameter to scan more records for columns [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #40124: [Python] from_pylist should allow a parameter to scan more records for columns
URL: https://github.com/apache/arrow/issues/40124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org