[I] Create testing functions for PyArrow.Table comparisons like `pandas.testing.assert_frame_equals()` [arrow]

["d33bs (via GitHub)" <gi...@apache.org>]

d33bs opened a new issue, #44530:
URL: https://github.com/apache/arrow/issues/44530

   ### Describe the enhancement requested
   
   Thanks for the great efforts surrounding this project and the surrounding ecosystem! I find that when I'm using large PyArrow Tables that I sometimes need to create data tests which compare multiple tables for equality. At the moment I use the `pa.Table.equals()` method to make these comparisons. The output from these comparisons during, for example, `pytest` testing can be very hard to read with large tables (hundreds of thousands of rows, thousands of columns) - it can be nearly impossible to find the differences unless you know exactly where to look. As a result, I find myself leaning on a conversion via `pa.Table.to_pandas()` and then leveraging `pandas.testing.assert_frame_equals()` to garner better output and an idea of where the data issues may lie. The conversion can be costly with larger datasets both in terms of time and accuracy (conversions and `assert_frame_equals` settings).
   
   It'd be a fantastic time saver to have a function or some sort of testing suite which is ready to use within PyArrow itself! Perhaps something like a `pa.testing.assert_table_equals()` which could help with repetitive tasks like this without the cost of conversion into another package's format (like Pandas). If something like this already exists within another side of the Arrow project it'd be great to share that capability within PyArrow somehow.
   
   Please let me know if I may clarify at all and thanks in advance for any thoughts!
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Create testing functions for PyArrow.Table comparisons like `pandas.testing.assert_frame_equals()` [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #44530:
URL: https://github.com/apache/arrow/issues/44530#issuecomment-3921260477

   This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Create testing functions for PyArrow.Table comparisons like `pandas.testing.assert_frame_equals()` [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #44530:
URL: https://github.com/apache/arrow/issues/44530#issuecomment-3547050109

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Create testing functions for PyArrow.Table comparisons like `pandas.testing.assert_frame_equals()` [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic closed issue #44530: Create testing functions for PyArrow.Table comparisons like `pandas.testing.assert_frame_equals()`
URL: https://github.com/apache/arrow/issues/44530


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org