[GitHub] [arrow] R-JunmingChen commented on issue #35268: [C++] OrderBy with spillover

["R-JunmingChen (via GitHub)" <gi...@apache.org>]

R-JunmingChen commented on issue #35268:
URL: https://github.com/apache/arrow/issues/35268#issuecomment-1576946749

   Hi, @westonpace, I am stuck with reading back data. I need to read back data with specific offset and batch size in spillover files.
   Currenty, my offset/batch size counts on the number of rows and I use parquet format for temporal spillover.
   However, our parquet lib doesn't support read file with row offset( only the byte offset are supported). Besides, the recordBatchReader, which is a generator, supports batch_size with the number of row but it doesn't support offset.
   
   May be AsyncGenerator is a good choice to utilize recordBatchReader, but I don't find examples for reading parquet file, the test cases of it are all memory operations. If the AsyncGenerator could resolve the problem, could you please show me a simple example?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org