You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "discivigour (via GitHub)" <gi...@apache.org> on 2026/07/14 15:11:21 UTC
[PR] [python] Optimize BLOB write performance [paimon]
discivigour opened a new pull request, #8625:
URL: https://github.com/apache/paimon/pull/8625
## What is the purpose of the change?
Improve Python BLOB write performance by reducing CRC and Arrow row-selection overhead.
## Brief change log
- Use `isal_zlib.crc32` when available, with `zlib` as the fallback.
- Reuse the original Arrow batch when all rows belong to one partition/bucket.
- Use zero-copy Arrow `slice` for contiguous partition/bucket groups.
- Keep Arrow `take` for non-contiguous groups.
- Add coverage for CRC compatibility and all three Arrow grouping paths.
## Performance
A 60 GB BLOB write test improved as follows:
| Data size | Before | After | Elapsed-time reduction |
| --- | ---: | ---: | ---: |
| 60 GB | 178.984 s | 132.871 s | 25.76% |
The elapsed time decreased by 46.113 seconds, from 178.984 seconds to 132.871 seconds.
## Tests
- Verify the fallback CRC written to BLOB records matches standard `zlib.crc32`.
- Verify `isal_zlib` and `zlib` produce byte-identical BLOB records.
- Verify full batches are passed through without Arrow `take`.
- Verify contiguous groups use zero-copy Arrow `slice`.
- Verify non-contiguous groups continue to use Arrow `take`.
- Verify partitioned BLOB data can be written and read end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Optimize BLOB write performance [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8625:
URL: https://github.com/apache/paimon/pull/8625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Optimize BLOB write performance [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8625:
URL: https://github.com/apache/paimon/pull/8625#issuecomment-4992893409
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Optimize BLOB write performance [paimon]
Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on code in PR #8625:
URL: https://github.com/apache/paimon/pull/8625#discussion_r3593602739
##########
paimon-python/pypaimon/write/blob_format_writer.py:
##########
@@ -16,9 +16,13 @@
# under the License.
import struct
-import zlib
from typing import BinaryIO, List, Optional
+try:
+ from isal import isal_zlib as crc_backend
Review Comment:
Done. The `isal` CRC compatibility test is now enabled and passes in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Optimize BLOB write performance [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8625:
URL: https://github.com/apache/paimon/pull/8625#discussion_r3592942243
##########
paimon-python/pypaimon/write/blob_format_writer.py:
##########
@@ -16,9 +16,13 @@
# under the License.
import struct
-import zlib
from typing import BinaryIO, List, Optional
+try:
+ from isal import isal_zlib as crc_backend
Review Comment:
Can you enable isal in github CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org