[I] Feather chunksize doesn't round-trip [arrow]

["alippai (via GitHub)" <gi...@apache.org>]

alippai opened a new issue, #45422:
URL: https://github.com/apache/arrow/issues/45422

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   I tried this with pyarrow 19:
   ```python
   import pyarrow.feather as pf
   t = ...
   pf.write_feather(t, 'test.feather', chunksize=1024*1024)
   len(pf.read_table('test.feather').to_batches()[0]) // 65k
   pf.write_feather(t, 'test2.feather', chunksize=256*1024)
   len(pf.read_table('test2.feather').to_batches()[0]) // 65k
   ```
   
   I expected the files to be different (different compressed sizes), but they are byte-by-byte identical. As a consequence the batch sizes are lost when reading the data back. 
   
   Do I assume correctly the file should consist of chunksize long buffers for each column (per recordbatch) and these buffers are independently compressed using lz4 or zstd?
   
   ### Component(s)
   
   Python, C++, Format


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Feather chunksize doesn't round-trip [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] closed issue #45422: [Python] Feather chunksize doesn't round-trip
URL: https://github.com/apache/arrow/issues/45422


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python] Feather chunksize doesn't round-trip [arrow]

["raulcd (via GitHub)" <gi...@apache.org>]

raulcd commented on issue #45422:
URL: https://github.com/apache/arrow/issues/45422#issuecomment-2640127744

   Hi @alippai ! Thanks for raising the issue. You might get a quicker response if you reach the user mailing list. I would recommend you to send and email to the user discussions: [user@arrow.apache.org](mailto:user@arrow.apache.org) (first subscribe by sending an e-mail to [user-subscribe@arrow.apache.org](mailto:user-subscribe@arrow.apache.org) if you are not already subscribed).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Feather chunksize doesn't round-trip [arrow]

["alippai (via GitHub)" <gi...@apache.org>]

alippai commented on issue #45422:
URL: https://github.com/apache/arrow/issues/45422#issuecomment-2635035677

   Is this the equivalent?
   ```python
   BATCH_SIZE = 1024*1024
   
   if len(t.to_batches) > 1:
      t = t.combine_chunks()
   with pa.OSFile('test3.feather', 'wb') as sink:
      with pa.ipc.new_file(sink, f2.schema, options=pa.ipc.IpcWriteOptions(compression='lz4')) as writer:
         for batch in t.to_batches(BATCH_SIZE):
            writer.write(batch)
   len(pf.read_table('test3.feather').to_batches()[0]) # 1024*1024 rows
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org