You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "cedriccuypers (via GitHub)" <gi...@apache.org> on 2024/12/12 10:35:34 UTC

[I] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

cedriccuypers opened a new issue, #45009:
URL: https://github.com/apache/arrow/issues/45009

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   We noticed a bug in pyarrow when we were trying to iterate in batches over parquet files, of which some had zero row groups (output of the AWS S3 Inventory service).
   
   Code snippet to reproduce the issue:
   
   ```
   import fastparquet
   import pandas as pd
   import pyarrow
   import pyarrow.parquet as pq
   
   print(f"Using pyarrow version {pyarrow.__version__}")
   
   df = pd.DataFrame({"a": pd.Series(dtype="int"), "b": pd.Series(dtype="str"), "c": pd.Series(dtype="float")})
   
   empty_parquet_file_path = "my_empty_parquet_file.parquet"
   
   fastparquet.write(empty_parquet_file_path, df, row_group_offsets=[])
   
   assert pq.read_metadata(empty_parquet_file_path).num_row_groups == 0
   
   parquet_file = pq.ParquetFile(empty_parquet_file_path)
   
   for batch in parquet_file.iter_batches():
       print(batch)
   ```
   
   The following error is raised when using pyarrow 18.0.0 or 18.1.0. 
   ```
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "pyarrow/_parquet.pyx", line 1634, in iter_batches
     File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
   OSError: The file only has 0 row groups, requested metadata for row group: -1
   ```
   
   In pyarrow 17, there is no issue, and an empty parquet file doesn't seem to produce any batches when calling iter_batches, which is the behaviour I would expect.
   
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

Posted by "brendanyounger (via GitHub)" <gi...@apache.org>.
brendanyounger commented on issue #45009:
URL: https://github.com/apache/arrow/issues/45009#issuecomment-2576255812

   I would like to second this. This is a bug in the core cpp parquet source code which, as the OP points out, was not present in any previous version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

Posted by "thras-deepcure (via GitHub)" <gi...@apache.org>.
thras-deepcure commented on issue #45009:
URL: https://github.com/apache/arrow/issues/45009#issuecomment-2655696176

   This issue persists with pyarrow 19.0.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Python] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #45009:
URL: https://github.com/apache/arrow/issues/45009#issuecomment-4780146955

   Confirming this is fixed in newer versions, most probably with https://github.com/apache/arrow/pull/45350.
   I have tested on latest main:
   
   ```python
   >>> import pyarrow.parquet as pq
   >>> import pyarrow as pa
   
   >>> pa.__version__
   '25.0.0.dev175+g8f3085e35.d20260623'
   
   >>> table = pa.table({'one': [-1, None, 2.5],
   ...                   'two': ['foo', 'bar', 'baz'],
   ...                   'three': [True, False, True]})
   
   >>> with pq.ParquetWriter('example_empty.parquet', table.schema) as writer:
   ...     pass
   ... 
   
   >>> parquet_file = pq.ParquetFile('example_empty.parquet')
   >>> for batch in parquet_file.iter_batches():
   ...     print(batch)
   ... 
   
   >>> parquet_file.num_row_groups
   0
   >>> list(parquet_file.iter_batches())
   []
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Python] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #45009: [C++][Python] iter_batches on a parquet file with zero row groups fails in pyarrow>=18
URL: https://github.com/apache/arrow/issues/45009


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

Posted by "mjtieman (via GitHub)" <gi...@apache.org>.
mjtieman commented on issue #45009:
URL: https://github.com/apache/arrow/issues/45009#issuecomment-2656884784

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] iter_batches on a parquet file with zero row groups fails in pyarrow>=18 [arrow]

Posted by "binaryaaron (via GitHub)" <gi...@apache.org>.
binaryaaron commented on issue #45009:
URL: https://github.com/apache/arrow/issues/45009#issuecomment-2682587780

   seems fixed in `v19.0.1`.
   
    https://github.com/apache/arrow/pull/43945/files/dcd3bdfd1ae313733c670567cd692016cb5523a9
    issue here: https://github.com/apache/arrow/issues/45339. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org