You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/06/07 23:17:31 UTC
[GitHub] [kafka] ijuma opened a new pull request #8830: KAFKA-10116: GraalVM native-image prototype
ijuma opened a new pull request #8830:
URL: https://github.com/apache/kafka/pull/8830
I can generate a native image with the added task, but it exists soon
after start-up.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787566289
@rspurgeon Thanks, I suspected the config files were not being picked up. I also pushed a classpath fix. I was able to produce and consume with this. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787580589
Resident memory starts at about 460 MB:
> ijuma 917472 73.2 0.7 5954368 472220 pts/0 Sl+ 17:48 0:15 ./core/kafka.kafka config/server.properties
Producer perf achieves 330 MB/s:
> [ijuma@desktop]~/src/kafka% ./bin/kafka-producer-perf-test.sh --producer.config config/producer.properties --topic=test-topic-16-partitions --num-records=20000000 --record-size=1000 --throughput=1000000000 --print-metrics
> 1492565 records sent, 298513.0 records/sec (284.68 MB/sec), 103.5 ms avg latency, 222.0 ms max latency.
> 1612080 records sent, 322416.0 records/sec (307.48 MB/sec), 101.5 ms avg latency, 223.0 ms max latency.
> 1651531 records sent, 330306.2 records/sec (315.00 MB/sec), 99.0 ms avg latency, 236.0 ms max latency.
> 1700573 records sent, 340114.6 records/sec (324.36 MB/sec), 96.3 ms avg latency, 250.0 ms max latency.
> 1740014 records sent, 348002.8 records/sec (331.88 MB/sec), 94.0 ms avg latency, 285.0 ms max latency.
> 1750641 records sent, 350128.2 records/sec (333.91 MB/sec), 93.3 ms avg latency, 267.0 ms max latency.
> 1759611 records sent, 351922.2 records/sec (335.62 MB/sec), 93.2 ms avg latency, 262.0 ms max latency.
> 1764414 records sent, 352882.8 records/sec (336.54 MB/sec), 92.8 ms avg latency, 266.0 ms max latency.
consumer-perf achieves 1 GB/s:
> [ijuma@desktop]~/src/kafka% ./bin/kafka-consumer-perf-test.sh --topic test-topic-16-partitions --consumer.config config/consumer.properties --messages 10000000 --broker-list localhost:9092 --show-detailed-stats
> time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
> 2021-02-28 17:00:20:066, 0, 5082.0122, 1016.4024, 5328876, 1065775.2000, 357, 4643, 1094.5536, 1147722.5932
The kafka process resident memory is ~500 MB at the end of it:
> ijuma 888772 1.7 0.7 5743068 512008 pts/0 Sl+ 16:58 0:46 ./core/kafka.kafka config/server.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787583534
@rspurgeon Agreed, that's the other metric I want to capture, but:
1. The shutdown issue you mentioned means that restarts are not clean (I have to kill -9 the process)
2. For some reason, I am not seeing the log messages I expect. Need to figure out why. Is it working correctly for you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-786989056
Rebased and regenerated the configs with graalvm ce 21.0.0.2. Need to investigate the error during start-up:
> Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class kafka.utils.Exit$
> at com.oracle.svm.core.classinitialization.ClassInitializationInfo.initialize(ClassInitializationInfo.java:239)
> at kafka.Kafka$.main(Kafka.scala:122)
> at kafka.Kafka.main(Kafka.scala)
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rspurgeon commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
rspurgeon commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787582318
Another interesting metric to capture could be startup time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rspurgeon commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
rspurgeon commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787584506
> @rspurgeon Agreed, that's the other metric I want to capture, but:
>
> 1. The shutdown issue you mentioned means that restarts are not clean (I have to kill -9 the process)
>
> 2. For some reason, I am not seeing the log messages I expect. Need to figure out why. Is it working correctly for you?
I don't get logging or clean shutdown either. I can try to look more in the coming days
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma edited a comment on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
ijuma edited a comment on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787580589
The generated binary is ~62 MB:
> [ijuma@desktop]~/src/kafka% ls -al core/kafka.kafka
> -rwxr-xr-x. 1 ijuma ijuma 65329368 Feb 28 15:55 core/kafka.kafka
Resident memory starts at about 460 MB:
> ijuma 917472 73.2 0.7 5954368 472220 pts/0 Sl+ 17:48 0:15 ./core/kafka.kafka config/server.properties
Producer perf achieves 330 MB/s:
> [ijuma@desktop]~/src/kafka% ./bin/kafka-producer-perf-test.sh --producer.config config/producer.properties --topic=test-topic-16-partitions --num-records=20000000 --record-size=1000 --throughput=1000000000 --print-metrics
> 1492565 records sent, 298513.0 records/sec (284.68 MB/sec), 103.5 ms avg latency, 222.0 ms max latency.
> 1612080 records sent, 322416.0 records/sec (307.48 MB/sec), 101.5 ms avg latency, 223.0 ms max latency.
> 1651531 records sent, 330306.2 records/sec (315.00 MB/sec), 99.0 ms avg latency, 236.0 ms max latency.
> 1700573 records sent, 340114.6 records/sec (324.36 MB/sec), 96.3 ms avg latency, 250.0 ms max latency.
> 1740014 records sent, 348002.8 records/sec (331.88 MB/sec), 94.0 ms avg latency, 285.0 ms max latency.
> 1750641 records sent, 350128.2 records/sec (333.91 MB/sec), 93.3 ms avg latency, 267.0 ms max latency.
> 1759611 records sent, 351922.2 records/sec (335.62 MB/sec), 93.2 ms avg latency, 262.0 ms max latency.
> 1764414 records sent, 352882.8 records/sec (336.54 MB/sec), 92.8 ms avg latency, 266.0 ms max latency.
consumer-perf achieves 1 GB/s:
> [ijuma@desktop]~/src/kafka% ./bin/kafka-consumer-perf-test.sh --topic test-topic-16-partitions --consumer.config config/consumer.properties --messages 10000000 --broker-list localhost:9092 --show-detailed-stats
> time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
> 2021-02-28 17:00:20:066, 0, 5082.0122, 1016.4024, 5328876, 1065775.2000, 357, 4643, 1094.5536, 1147722.5932
The kafka process resident memory is ~500 MB at the end of it:
> ijuma 888772 1.7 0.7 5743068 512008 pts/0 Sl+ 16:58 0:46 ./core/kafka.kafka config/server.properties
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rspurgeon commented on pull request #8830: KAFKA-10116: GraalVM native-image prototype
Posted by GitBox <gi...@apache.org>.
rspurgeon commented on pull request #8830:
URL: https://github.com/apache/kafka/pull/8830#issuecomment-787511695
@ijuma I'm following your work here. Just thought I'd share what I see.
I was able to get a functioning native-image broker by putting the native-image config files in the resources folder in the core project:
```
core/src/main/resources/META-INF/native-image
drwxr-xr-x - rspurgeon 28 Feb 13:32 core/src/main/resources/META-INF/native-image
.rw-r--r-- 1.2k rspurgeon 28 Feb 13:39 ├── jni-config.json
.rw-r--r-- 33 rspurgeon 28 Feb 13:39 ├── proxy-config.json
.rw-r--r-- 11k rspurgeon 28 Feb 13:39 ├── reflect-config.json
.rw-r--r-- 177 rspurgeon 28 Feb 13:39 ├── resource-config.json
.rw-r--r-- 166 rspurgeon 28 Feb 13:39 └── serialization-config.json
```
and then running:
`./gradlew clean core:nativeImage`
This ran for me:
```
./core/kafka.kafka ./config/raft-combined.properties
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
```
and I could connect a client to list topics. I do get a SIGINT exception when trying to shutdown the broker from the cli with ctrl-c.
I also turned on --verbose for native-image so I could verify it was reading the configs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org