[GitHub] [arrow] shner-elmo opened a new issue, #36865: [Python] Instantiate `pa.Table` from a `Generator`/`Iterator`

["shner-elmo (via GitHub)" <gi...@apache.org>]

shner-elmo opened a new issue, #36865:
URL: https://github.com/apache/arrow/issues/36865

   ### Describe the enhancement requested
   
   First of all thanks for the great project which Pyarrow is.
   
   When I attempt to create a table from a list it works fine, but when I create it from a generator it complains:
   ```py
   >>> data: list[dict]
   >>> table = pa.Table.from_pylist(data, schema=SCHEMA)
   >>> table = pa.Table.from_pylist((dct for dct in data), schema=SCHEMA)
   Traceback (most recent call last):
     File "/home/shner/.cache/pypoetry/virtualenvs/ame-wikipedia-pipeline-FN8t627X-py3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3508, in run_code
       exec(code_obj, self.user_global_ns, self.user_ns)
     File "<ipython-input-15-87d9c5bc5106>", line 1, in <module>
       table = pa.Table.from_pylist((dct for dct in data), schema=SCHEMA)
     File "pyarrow/table.pxi", line 3906, in pyarrow.lib.Table.from_pylist
     File "pyarrow/table.pxi", line 5460, in pyarrow.lib._from_pylist
     File "pyarrow/table.pxi", line 3798, in pyarrow.lib.Table.from_arrays
     File "pyarrow/table.pxi", line 2962, in pyarrow.lib.Table.validate
     File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
   pyarrow.lib.ArrowInvalid: Column 1 named year expected length 389612 but got length 0
   ```
   I guess that kind of is expected since the method is called `from_pylist()`, but usually most functions that take in some data will accept more general types, it would be great if you guys could extend this to accept: `Iterable[dict[str, ...]] | Iterator[dict[str, ...]]`, 
   or at least to specify which methods and properties the data structure should have, for example: `SupportslenAndGetitem`, `typing.Collection` or `typing.Sequence` and so on.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Instantiate `pa.Table` from a `Generator`/`Iterator` [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic closed issue #36865: [Python] Instantiate `pa.Table` from a `Generator`/`Iterator`
URL: https://github.com/apache/arrow/issues/36865


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Instantiate `pa.Table` from a `Generator`/`Iterator` [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #36865:
URL: https://github.com/apache/arrow/issues/36865#issuecomment-3921779936

   This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org