You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lwpyr (via GitHub)" <gi...@apache.org> on 2023/06/13 18:19:13 UTC

[GitHub] [arrow-rs] lwpyr opened a new issue, #4410: Wrongly calculated data compressed length in IPC writer

lwpyr opened a new issue, #4410:
URL: https://github.com/apache/arrow-rs/issues/4410

   **Describe the bug**
   The wrongly calculated compressed length included the full original buffer length, which will decline almost all the compressable data.
   Suppose original buffer len is *a*, incoming data len is *b*, compressed data len is *c*, the code should compare *b* and *c* instead of *b* and *a+c*
   
   **To Reproduce**
   1. Use Python to write a IPC file compressed with ZSTD, the size is 1.7 MB
   2. Use arrow-rs to read data from python encoded IPC file, then write again to the disk with ZSTD encoded, but the file size is 20 MB.
   3. Apply the change of the PR, the file size produced by arrow-rs is 1.6 MB
   
   **Expected behavior**
   arrow-rs IPC writer should compress those data worth compression.
   
   **Additional context**
   N/A


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb commented on issue #4410: Wrongly calculated data compressed length in IPC writer

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #4410:
URL: https://github.com/apache/arrow-rs/issues/4410#issuecomment-1595001739

   `label_issue.py` automatically added labels {'arrow'} from #4411


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb closed issue #4410: Wrongly calculated data compressed length in IPC writer

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #4410: Wrongly calculated data compressed length in IPC writer
URL: https://github.com/apache/arrow-rs/issues/4410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org