You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/06/09 03:20:27 UTC
[GitHub] [kafka] dchristle opened a new pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
dchristle opened a new pull request #10847:
URL: https://github.com/apache/kafka/pull/10847
This PR aims to upgrade `zstd-jni` from `1.4.9-1` to `1.5.0-1`.
This change will incorporate a number of bug fixes and performance improvements made in the `1.4.x` branch and `1.5.0` of `zstd`:
- https://github.com/facebook/zstd/releases/tag/v1.5.0
- https://github.com/facebook/zstd/releases/tag/v1.4.9
- https://github.com/facebook/zstd/releases/tag/v1.4.8
- https://github.com/facebook/zstd/releases/tag/v1.4.7
- https://github.com/facebook/zstd/releases/tag/v1.4.5
- https://github.com/facebook/zstd/releases/tag/v1.4.4
- https://github.com/facebook/zstd/releases/tag/v1.4.3
- https://github.com/facebook/zstd/releases/tag/v1.4.2
The most recent `1.5.0` release offers +25%-140% (compression) +15% (decompression) performance improvements.
Since this is a dependency change, this should pass all the existing CIs.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dongjinleekr commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dongjinleekr commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-859370741
@ijuma @dchristle I am now evaluating Zstandard's long window mode. For details, please check [the updated KIP-390](https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma merged pull request #10847: KAFKA-12921: Upgrade ZSTD JNI to 1.5.0-2
Posted by GitBox <gi...@apache.org>.
ijuma merged pull request #10847:
URL: https://github.com/apache/kafka/pull/10847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dchristle edited a comment on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dchristle edited a comment on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858843145
> The current version is 1.4.9, so I'm a bit confused why we're mentioning anything besides 1.5.0.
Woops - I'm getting my wires crossed on a different zstd 1.5.0 related PR I have with a larger upgrade. You are right -- this is just from `1.4.9-1` to `1.5.0-1`. Sorry for my confusion. I updated the PR description to reflect this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dchristle commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dchristle commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-857396796
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858615783
Also, why are we listing versions in the PR description that are not relevant to this upgrade?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dchristle commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dchristle commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858843145
> The current version is 1.4.9, so I'm a bit confused why we're mentioning anything besides 1.5.0.
Woops - I'm getting my wires crossed on a different zstd 1.5.0 related PR I have with a larger upgrade. You are right -- this is just from `1.4.9-1` to `1.5.0-1`. Sorry for my confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dongjinleekr commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dongjinleekr commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858653169
@ijuma @dchristle Since we have more time for [KIP-390](https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Allow+fine-grained+configuration+for+compression), I will run the benchmark with this zstd binding. Stay tuned!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma edited a comment on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
ijuma edited a comment on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858829828
The current version is 1.4.9, so I'm a bit confused why we're mentioning anything besides 1.5.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858829828
The current version if 1.4.9, so I'm a bit confused why we're mentioning anything besides 1.5.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858614341
This is a good change, but can we please quality the perf improvements claim? My understanding is that only applies to certain compression levels and Kafka currently always picks a specific one. @dongjinleekr is working on making that configurable via a separate KIP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI to 1.5.0-2
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-860235280
Unrelated flaky failures:
> Build / JDK 11 and Scala 2.13 / org.apache.kafka.streams.integration.SmokeTestDriverIntegrationTest.shouldWorkWithRebalance
> Build / JDK 8 and Scala 2.12 / org.apache.kafka.streams.integration.SmokeTestDriverIntegrationTest.shouldWorkWithRebalance
> Build / JDK 8 and Scala 2.12 / org.apache.kafka.streams.integration.SmokeTestDriverIntegrationTest.shouldWorkWithRebalance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dchristle edited a comment on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dchristle edited a comment on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858804912
@ijuma
> This is a good change, but can we please quality the perf improvements claim? My understanding is that only applies to certain compression levels and Kafka currently always picks a specific one. @dongjinleekr is working on making that configurable via a separate KIP.
It is true that the most recent performance improvements I quoted (for `1.5.0`) appear only in mid-range compression levels.
> Also, why are we listing versions in the PR description that are not relevant to this upgrade?
I tried to follow a previous `zstd-jni` PR's convention here: https://github.com/apache/kafka/pull/10285 . I think it gives context on the magnitude of the upgrade, but I can change the commit message/PR title to remove the existing version reference if you like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dchristle commented on pull request #10847: KAFKA-12921: Upgrade ZSTD JNI from 1.4.9-1 to 1.5.0-1
Posted by GitBox <gi...@apache.org>.
dchristle commented on pull request #10847:
URL: https://github.com/apache/kafka/pull/10847#issuecomment-858804912
@ijuma
> This is a good change, but can we please quality the perf improvements claim? My understanding is that only applies to certain compression levels and Kafka currently always picks a specific one. @dongjinleekr is working on making that configurable via a separate KIP.
It is true that the most recent performance improvements I quoted (for `1.5.0`) appear only in mid-range compression levels. I did not highlight it in my description, but besides bug fixes, the earlier releases quote consistent perf improvements:
- `1.4.4`: ~10% decompression bump, no level-dependence quoted
- `1.4.5`: 5-10% decompression improvement in `x86_64` architecture, +15-50% in various `arm` processors
- `1.4.7`: Improved `--long` mode compression ratio at high levels, 5-30% decompression improvement for blocks < 32kB
- `1.4.9`: 2x faster `--long` mode compression speed
> Also, why are we listing versions in the PR description that are not relevant to this upgrade?
I tried to follow a previous `zstd-jni` PR's convention here: https://github.com/apache/kafka/pull/10285 . I think it gives context on the magnitude of the upgrade, but I can change the commit message/PR title to remove the existing version reference if you like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org