You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2024/08/09 13:18:22 UTC

[I] Error on dataframe count using arrow dataset [datafusion-python]

timsaucer opened a new issue, #800:
URL: https://github.com/apache/datafusion-python/issues/800

   **Describe the bug**
   When using a `pyarrow.dataset` as your source and performing a dataframe `count` operation you get an error.
   
   **To Reproduce**
   You can point the below snippet to any parquet file.
   
   ```
   from datafusion import SessionContext
   import pyarrow.dataset as ds
   
   ctx = SessionContext()
   file_path = "/some-path/datafusion-python/examples/tpch/data/lineitem.parquet"
   pyarrow_dataset = ds.dataset([file_path])
   
   ctx.register_dataset("pyarrow_dataset", pyarrow_dataset)
   df = ctx.table("pyarrow_dataset").select("l_orderkey", "l_partkey", "l_linenumber")
   
   df.limit(3).show()
   df.count()
   ```
   
   This generates the following output. The `show` is to demonstrate the file is read appropriately.
   
   ```
   DataFrame()
   +------------+-----------+--------------+
   | l_orderkey | l_partkey | l_linenumber |
   +------------+-----------+--------------+
   | 1          | 155190    | 1            |
   | 1          | 67310     | 2            |
   | 1          | 63700     | 3            |
   +------------+-----------+--------------+
   Traceback (most recent call last):
     File "/Users/tsaucer/src/personal/arrow_rs_dataset_read/count_dataset_read.py", line 16, in <module>
       df.count()
     File "/Users/tsaucer/src/personal/datafusion-python/python/datafusion/dataframe.py", line 507, in count
       return self.df.count()
              ^^^^^^^^^^^^^^^
   Exception: External error: Arrow error: External error: ArrowException: Invalid argument error: must either specify a row count or at least one column
   ```
   **Expected behavior**
   `count()` should return the number of rows in this dataset.
   
   Work around is to aggregate and count
   ```
   from datafusion import col, functions as f
   df.aggregate([], [f.count(col("l_orderkey"))]).show()
   ```
   
   **Additional context**
   In my investigation, I found that we register arrow datasets by creating a `TableProvider` in `src/dataset.rs` and then the execution calls happen in `src/dataset_exec.rs`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Error on dataframe count using arrow dataset [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #800:
URL: https://github.com/apache/datafusion-python/issues/800#issuecomment-2277935767

   I cannot reproduce this in the `datafusion` repo because `arrow-rs` doesn't appear to have the concept of dataset in the same way as `pyarrow` that I can find.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Error on dataframe count using arrow dataset [datafusion-python]

Posted by "Michael-J-Ward (via GitHub)" <gi...@apache.org>.
Michael-J-Ward commented on issue #800:
URL: https://github.com/apache/datafusion-python/issues/800#issuecomment-2313438334

   I tracked down the error.
   
   It occurs when attempting to convert the `pyarrow` result into an `arrow-rs` `RecordBatch`
   
   https://github.com/apache/datafusion-python/blob/ae7470e99cad4650675ea361aebc64da78dd1f39/src/dataset_exec.rs#L61-L65
   
   `arrow-rs` does have an option for creating "count" like `RecordBatches` but it requires an additional config in `RecordBatchOptions`: https://github.com/apache/arrow-rs/pull/1552
   
   However, the `from_pyarrow_bound` method only uses the default options.
   https://github.com/apache/arrow-rs/blob/b711f23a136e0b094a70a4aafb020d4bb9f60619/arrow/src/pyarrow.rs#L334-L392
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Error on dataframe count using arrow dataset [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #800: Error on dataframe count using arrow dataset
URL: https://github.com/apache/datafusion-python/issues/800


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org