You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@openwhisk.apache.org by GitBox <gi...@apache.org> on 2021/02/01 01:00:03 UTC

[GitHub] [openwhisk] ningyougang opened a new pull request #5046: Make kafka version configurable

ningyougang opened a new pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   <!--- Provide a concise summary of your changes in the Title -->
   
   - [x] Make kafka version configurable
   - [x] Make kafka version using latest version: `2.7.0` 
   
   kafka latest version: `2.7.0` fixed a bug: https://issues.apache.org/jira/browse/KAFKA-8334
   
   It happened in our env, after deploy latest kafka version in PM, the problem gone.
   ## Description
   <!--- Provide a detailed description of your changes. -->
   <!--- Include details of what problem you are solving and how your changes are tested. -->
   
   ## Related issue and scope
   <!--- Please include a link to a related issue if there is one. -->
   - [ ] I opened an issue to propose and discuss this change (#????)
   
   ## My changes affect the following components
   <!--- Select below all system components are affected by your change. -->
   <!--- Enter an `x` in all applicable boxes. -->
   - [ ] API
   - [ ] Controller
   - [x] Message Bus (e.g., Kafka)
   - [ ] Loadbalancer
   - [ ] Invoker
   - [ ] Intrinsic actions (e.g., sequences, conductors)
   - [ ] Data stores (e.g., CouchDB)
   - [ ] Tests
   - [ ] Deployment
   - [ ] CLI
   - [ ] General tooling
   - [ ] Documentation
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Use `x` in all the boxes that apply: -->
   - [x] Bug fix (generally a non-breaking change which closes an issue).
   - [ ] Enhancement or new feature (adds new functionality).
   - [ ] Breaking change (a bug fix or enhancement which changes existing behavior).
   
   ## Checklist:
   <!--- Please review the points below which help you make sure you've covered all aspects of the change you're making. -->
   
   - [x] I signed an [Apache CLA](https://github.com/apache/openwhisk/blob/master/CONTRIBUTING.md).
   - [x] I reviewed the [style guides](https://github.com/apache/openwhisk/wiki/Contributing:-Git-guidelines#code-readiness) and followed the recommendations (Travis CI will check :).
   - [ ] I added tests to cover my changes.
   - [ ] My changes require further changes to the documentation.
   - [ ] I updated the documentation where necessary.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] codecov-io edited a comment on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-762121973


   # [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=h1) Report
   > Merging [#5046](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=desc) (5cde872) into [master](https://codecov.io/gh/apache/openwhisk/commit/66868205b52ee65f28756038c44d8df5b96d2bcc?el=desc) (6686820) will **decrease** coverage by `7.43%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/openwhisk/pull/5046/graphs/tree.svg?width=650&height=150&src=pr&token=l0YmsiSAso)](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #5046      +/-   ##
   ==========================================
   - Coverage   82.35%   74.91%   -7.44%     
   ==========================================
     Files         206      206              
     Lines       10006    10006              
     Branches      445      445              
   ==========================================
   - Hits         8240     7496     -744     
   - Misses       1766     2510     +744     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...core/database/cosmosdb/RxObservableImplicits.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvUnhPYnNlcnZhYmxlSW1wbGljaXRzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ore/database/cosmosdb/cache/CacheInvalidator.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3Iuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...e/database/cosmosdb/cache/ChangeFeedConsumer.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NoYW5nZUZlZWRDb25zdW1lci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...core/database/cosmosdb/CosmosDBArtifactStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJBcnRpZmFjdFN0b3JlLnNjYWxh) | `0.00% <0.00%> (-95.85%)` | :arrow_down: |
   | [...sk/core/database/cosmosdb/CosmosDBViewMapper.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJWaWV3TWFwcGVyLnNjYWxh) | `0.00% <0.00%> (-93.90%)` | :arrow_down: |
   | [...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3JDb25maWcuc2NhbGE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...enwhisk/connector/kafka/KamonMetricsReporter.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYW1vbk1ldHJpY3NSZXBvcnRlci5zY2FsYQ==) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...scheduler/queue/ElasticSearchDurationChecker.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvRWxhc3RpY1NlYXJjaER1cmF0aW9uQ2hlY2tlci5zY2FsYQ==) | `0.00% <0.00%> (-78.85%)` | :arrow_down: |
   | [...e/database/cosmosdb/cache/KafkaEventProducer.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0thZmthRXZlbnRQcm9kdWNlci5zY2FsYQ==) | `0.00% <0.00%> (-78.58%)` | :arrow_down: |
   | [...whisk/core/database/cosmosdb/CosmosDBSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJTdXBwb3J0LnNjYWxh) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
   | ... and [28 more](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=footer). Last update [6686820...5cde872](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] codecov-io edited a comment on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-762121973


   # [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=h1) Report
   > Merging [#5046](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=desc) (5cde872) into [master](https://codecov.io/gh/apache/openwhisk/commit/66868205b52ee65f28756038c44d8df5b96d2bcc?el=desc) (6686820) will **decrease** coverage by `7.23%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/openwhisk/pull/5046/graphs/tree.svg?width=650&height=150&src=pr&token=l0YmsiSAso)](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #5046      +/-   ##
   ==========================================
   - Coverage   82.52%   75.28%   -7.24%     
   ==========================================
     Files         206      207       +1     
     Lines       10006    10202     +196     
     Branches      445      468      +23     
   ==========================================
   - Hits         8257     7681     -576     
   - Misses       1749     2521     +772     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...core/database/cosmosdb/RxObservableImplicits.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvUnhPYnNlcnZhYmxlSW1wbGljaXRzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ore/database/cosmosdb/cache/CacheInvalidator.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3Iuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...e/database/cosmosdb/cache/ChangeFeedConsumer.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NoYW5nZUZlZWRDb25zdW1lci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...core/database/cosmosdb/CosmosDBArtifactStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJBcnRpZmFjdFN0b3JlLnNjYWxh) | `0.00% <0.00%> (-95.85%)` | :arrow_down: |
   | [...sk/core/database/cosmosdb/CosmosDBViewMapper.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJWaWV3TWFwcGVyLnNjYWxh) | `0.00% <0.00%> (-93.90%)` | :arrow_down: |
   | [...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3JDb25maWcuc2NhbGE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...enwhisk/connector/kafka/KamonMetricsReporter.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYW1vbk1ldHJpY3NSZXBvcnRlci5zY2FsYQ==) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...scheduler/queue/ElasticSearchDurationChecker.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9zY2hlZHVsZXIvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9vcGVud2hpc2svY29yZS9zY2hlZHVsZXIvcXVldWUvRWxhc3RpY1NlYXJjaER1cmF0aW9uQ2hlY2tlci5zY2FsYQ==) | `0.00% <0.00%> (-78.85%)` | :arrow_down: |
   | [...e/database/cosmosdb/cache/KafkaEventProducer.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0thZmthRXZlbnRQcm9kdWNlci5zY2FsYQ==) | `0.00% <0.00%> (-78.58%)` | :arrow_down: |
   | [...whisk/core/database/cosmosdb/CosmosDBSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJTdXBwb3J0LnNjYWxh) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
   | ... and [28 more](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=footer). Last update [6686820...5cde872](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] jiangpengcheng merged pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
jiangpengcheng merged pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] rabbah commented on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
rabbah commented on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-769806527


   Thanks @ningyougang and @style95!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] style95 commented on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-768847272


   What I mentioned was the benchmarking of OpenWhisk with this version of Kafka.
   But I think it looks good to me as is.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] style95 edited a comment on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
style95 edited a comment on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-768847272


   What I mentioned was the benchmarking of OpenWhisk with this version of Kafka.
   But anyway it looks good to me as is.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang commented on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang commented on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-768798431


   Have any comment?
   
   I benchmarked against the `wurstmeister/kafka:2.13-2.7.0 `, worked well. result as below
   ### Env
   Env:  3 PM nodes（128G memory; 40 core cpu;）
   kafka version: `wurstmeister/kafka:2.13-2.7.0 `
   
   ### Benchmark test
   I run producer and consumer side's script at the same time in kafka0 container.
   * Producer side
   ```
   # send 50,000,000 records
   /opt/kafka_2.13-2.7.0/bin/kafka-producer-perf-test.sh --topic scheduler0 --num-records 50000000  --record-size 100  --throughput -1 --producer-props acks=0 bootstrap.servers=10.105.65.214:9093,10.105.65.215:9094,10.105.65.216:9095
   4077415 records sent, 815483.0 records/sec (77.77 MB/sec), 0.5 ms avg latency, 407.0 ms max latency.
   4763549 records sent, 952709.8 records/sec (90.86 MB/sec), 0.3 ms avg latency, 15.0 ms max latency.
   4392536 records sent, 878507.2 records/sec (83.78 MB/sec), 0.3 ms avg latency, 17.0 ms max latency.
   4739980 records sent, 947996.0 records/sec (90.41 MB/sec), 0.3 ms avg latency, 14.0 ms max latency.
   4678736 records sent, 935747.2 records/sec (89.24 MB/sec), 0.3 ms avg latency, 10.0 ms max latency.
   4289005 records sent, 857801.0 records/sec (81.81 MB/sec), 0.4 ms avg latency, 6.0 ms max latency.
   4657720 records sent, 931544.0 records/sec (88.84 MB/sec), 0.3 ms avg latency, 13.0 ms max latency.
   4393393 records sent, 878678.6 records/sec (83.80 MB/sec), 0.3 ms avg latency, 11.0 ms max latency.
   4286190 records sent, 857238.0 records/sec (81.75 MB/sec), 0.3 ms avg latency, 5.0 ms max latency.
   4440910 records sent, 888182.0 records/sec (84.70 MB/sec), 0.3 ms avg latency, 11.0 ms max latency.
   4519094 records sent, 903818.8 records/sec (86.19 MB/sec), 0.3 ms avg latency, 13.0 ms max latency.
   50000000 records sent, 897311.654284 records/sec (85.57 MB/sec), 0.32 ms avg latency, 407.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 10 ms 99.9th.
   ```
   I configured ` --throughput -1` to disable throttling, we can see, the tps is about `897,311`
   * Consumer side
   ```
   # Fetch 5,000,000  record (running for about 3 seconds)
   /opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 5000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
   start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
   2021-01-28 11:46:58:466, 2021-01-28 11:47:01:299, 476.8650, 168.3251, 5000292, 1765016.5902, 1611802018934, -1611802016101, -0.0000, -0.0031
   
   # Fetch 10,000,000   records  (running for about 5 seconds)
   /opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 10000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
   start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
   2021-01-28 11:45:58:096, 2021-01-28 11:46:03:224, 953.6926, 185.9775, 10000192, 1950115.4446, 1611801958563, -1611801953435, -0.0000, -0.0062
   
   # Fetch 1,000,000,000 records (running for about 20 minutes)
   /opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 1000000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
   start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
   2021-01-28 11:55:33:186, 2021-01-28 12:16:03:855, 95367.4615, 77.4924, 1000000313, 812566.4277, 1611802533652, -1611801302983, -0.0001, -0.6204
   ```
   We can saw the the tps is very high as well, when fetch `5,000,000`, the tps is `1,765,016`, when fetch `10,000,000` the tps is `1,950,115`, when fetch `1,000,000,000`, the tps is `81,2566`
   
   I also tested below scenes
   when running kafka-producer-perf-test.sh, i adjusted the `--num-records` to huge value to make the producer script running for long time, when running kafka-consumer-perf-test.sh, i ajusted the `--messages` to huge value as well to make the consumer script running for long time
   
   During benchmarking, i observed the cpu/memory doesn't consume too much for all kafka nodes.
   ![image](https://media.oss.navercorp.com/user/2324/files/19a21100-6158-11eb-8a50-79c02befa4f8)
   ![image](https://media.oss.navercorp.com/user/2324/files/03945080-6158-11eb-8fee-3c673b878158)
   ![image](https://media.oss.navercorp.com/user/2324/files/0db64f00-6158-11eb-9900-9e146e6d01b2)
   
   
   FYI, on our production env, the consumer.commit offset timeout(>=5.seconds) issue didn't reappeared after we used the latest kafka version.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] codecov-io commented on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-762121973


   # [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=h1) Report
   > Merging [#5046](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=desc) (862df50) into [master](https://codecov.io/gh/apache/openwhisk/commit/b0baa7b3c2aeff56fda0b826749e25df7067242a?el=desc) (b0baa7b) will **increase** coverage by `35.26%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/openwhisk/pull/5046/graphs/tree.svg?width=650&height=150&src=pr&token=l0YmsiSAso)](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #5046       +/-   ##
   ===========================================
   + Coverage   35.98%   71.24%   +35.26%     
   ===========================================
     Files         202      199        -3     
     Lines        9820     9735       -85     
     Branches      416      435       +19     
   ===========================================
   + Hits         3534     6936     +3402     
   + Misses       6286     2799     -3487     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...che/openwhisk/core/invoker/LogStoreCollector.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9Mb2dTdG9yZUNvbGxlY3Rvci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...a/org/apache/openwhisk/http/BasicHttpService.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvQmFzaWNIdHRwU2VydmljZS5zY2FsYQ==) | `3.33% <0.00%> (-86.67%)` | :arrow_down: |
   | [...la/org/apache/openwhisk/http/BasicRasService.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2h0dHAvQmFzaWNSYXNTZXJ2aWNlLnNjYWxh) | `16.66% <0.00%> (-83.34%)` | :arrow_down: |
   | [...pache/openwhisk/core/invoker/InvokerReactive.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9JbnZva2VyUmVhY3RpdmUuc2NhbGE=) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
   | [...la/org/apache/openwhisk/core/invoker/Invoker.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvaW52b2tlci9JbnZva2VyLnNjYWxh) | `1.58% <0.00%> (-68.26%)` | :arrow_down: |
   | [...apache/openwhisk/core/ack/MessagingActiveAck.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvYWNrL01lc3NhZ2luZ0FjdGl2ZUFjay5zY2FsYQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
   | [...whisk/connector/kafka/KafkaMessagingProvider.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYWZrYU1lc3NhZ2luZ1Byb3ZpZGVyLnNjYWxh) | `29.26% <0.00%> (-41.47%)` | :arrow_down: |
   | [...tainerpool/docker/DockerClientWithFileAccess.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9kb2NrZXIvRG9ja2VyQ2xpZW50V2l0aEZpbGVBY2Nlc3Muc2NhbGE=) | `56.00% <0.00%> (-36.00%)` | :arrow_down: |
   | [.../containerpool/docker/DockerContainerFactory.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9pbnZva2VyL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvY29udGFpbmVycG9vbC9kb2NrZXIvRG9ja2VyQ29udGFpbmVyRmFjdG9yeS5zY2FsYQ==) | `36.36% <0.00%> (-33.34%)` | :arrow_down: |
   | [...whisk/connector/kafka/KafkaProducerConnector.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYWZrYVByb2R1Y2VyQ29ubmVjdG9yLnNjYWxh) | `27.50% <0.00%> (-32.50%)` | :arrow_down: |
   | ... and [145 more](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=footer). Last update [b0baa7b...862df50](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] style95 commented on a change in pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
style95 commented on a change in pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#discussion_r559376290



##########
File path: ansible/group_vars/all
##########
@@ -150,7 +150,7 @@ kafka:
     protocols:
     - TLSv1.2
   protocol: "{{ kafka_protocol_for_setup }}"
-  version: 2.12-2.3.1
+  version: "{{ kafka_version | default('2.13-2.7.0') }}"

Review comment:
       Need to see if it does not introduce any regression.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] codecov-io edited a comment on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-762121973


   # [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=h1) Report
   > Merging [#5046](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=desc) (5cde872) into [master](https://codecov.io/gh/apache/openwhisk/commit/66868205b52ee65f28756038c44d8df5b96d2bcc?el=desc) (6686820) will **decrease** coverage by `7.34%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/openwhisk/pull/5046/graphs/tree.svg?width=650&height=150&src=pr&token=l0YmsiSAso)](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #5046      +/-   ##
   ==========================================
   - Coverage   82.52%   75.17%   -7.35%     
   ==========================================
     Files         206      211       +5     
     Lines       10006    10454     +448     
     Branches      445      471      +26     
   ==========================================
   - Hits         8257     7859     -398     
   - Misses       1749     2595     +846     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...core/database/cosmosdb/RxObservableImplicits.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvUnhPYnNlcnZhYmxlSW1wbGljaXRzLnNjYWxh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ore/database/cosmosdb/cache/CacheInvalidator.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3Iuc2NhbGE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...e/database/cosmosdb/cache/ChangeFeedConsumer.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NoYW5nZUZlZWRDb25zdW1lci5zY2FsYQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...core/database/cosmosdb/CosmosDBArtifactStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJBcnRpZmFjdFN0b3JlLnNjYWxh) | `0.00% <0.00%> (-95.85%)` | :arrow_down: |
   | [...sk/core/database/cosmosdb/CosmosDBViewMapper.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJWaWV3TWFwcGVyLnNjYWxh) | `0.00% <0.00%> (-93.90%)` | :arrow_down: |
   | [...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0NhY2hlSW52YWxpZGF0b3JDb25maWcuc2NhbGE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...enwhisk/connector/kafka/KamonMetricsReporter.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2Nvbm5lY3Rvci9rYWZrYS9LYW1vbk1ldHJpY3NSZXBvcnRlci5zY2FsYQ==) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...e/database/cosmosdb/cache/KafkaEventProducer.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29yZS9jb3Ntb3NkYi9jYWNoZS1pbnZhbGlkYXRvci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL29wZW53aGlzay9jb3JlL2RhdGFiYXNlL2Nvc21vc2RiL2NhY2hlL0thZmthRXZlbnRQcm9kdWNlci5zY2FsYQ==) | `0.00% <0.00%> (-78.58%)` | :arrow_down: |
   | [...whisk/core/database/cosmosdb/CosmosDBSupport.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvY29zbW9zZGIvQ29zbW9zREJTdXBwb3J0LnNjYWxh) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
   | [...ore/database/azblob/AzureBlobAttachmentStore.scala](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree#diff-Y29tbW9uL3NjYWxhL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvb3BlbndoaXNrL2NvcmUvZGF0YWJhc2UvYXpibG9iL0F6dXJlQmxvYkF0dGFjaG1lbnRTdG9yZS5zY2FsYQ==) | `11.53% <0.00%> (-60.58%)` | :arrow_down: |
   | ... and [25 more](https://codecov.io/gh/apache/openwhisk/pull/5046/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=footer). Last update [6686820...5cde872](https://codecov.io/gh/apache/openwhisk/pull/5046?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang edited a comment on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang edited a comment on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-768798431


   Have any comment?
   
   I benchmarked against the `wurstmeister/kafka:2.13-2.7.0 `, worked well. result as below
   ### Env
   Env:  3 PM nodes（128G memory; 40 core cpu;）
   kafka version: `wurstmeister/kafka:2.13-2.7.0 `
   
   ### Benchmark test
   I run producer and consumer side's script at the same time in kafka0 container.
   * Producer side
   ```
   # send 50,000,000 records
   /opt/kafka_2.13-2.7.0/bin/kafka-producer-perf-test.sh --topic scheduler0 --num-records 50000000  --record-size 100  --throughput -1 --producer-props acks=0 bootstrap.servers=10.105.65.214:9093,10.105.65.215:9094,10.105.65.216:9095
   4077415 records sent, 815483.0 records/sec (77.77 MB/sec), 0.5 ms avg latency, 407.0 ms max latency.
   4763549 records sent, 952709.8 records/sec (90.86 MB/sec), 0.3 ms avg latency, 15.0 ms max latency.
   4392536 records sent, 878507.2 records/sec (83.78 MB/sec), 0.3 ms avg latency, 17.0 ms max latency.
   4739980 records sent, 947996.0 records/sec (90.41 MB/sec), 0.3 ms avg latency, 14.0 ms max latency.
   4678736 records sent, 935747.2 records/sec (89.24 MB/sec), 0.3 ms avg latency, 10.0 ms max latency.
   4289005 records sent, 857801.0 records/sec (81.81 MB/sec), 0.4 ms avg latency, 6.0 ms max latency.
   4657720 records sent, 931544.0 records/sec (88.84 MB/sec), 0.3 ms avg latency, 13.0 ms max latency.
   4393393 records sent, 878678.6 records/sec (83.80 MB/sec), 0.3 ms avg latency, 11.0 ms max latency.
   4286190 records sent, 857238.0 records/sec (81.75 MB/sec), 0.3 ms avg latency, 5.0 ms max latency.
   4440910 records sent, 888182.0 records/sec (84.70 MB/sec), 0.3 ms avg latency, 11.0 ms max latency.
   4519094 records sent, 903818.8 records/sec (86.19 MB/sec), 0.3 ms avg latency, 13.0 ms max latency.
   50000000 records sent, 897311.654284 records/sec (85.57 MB/sec), 0.32 ms avg latency, 407.00 ms max latency, 0 ms 50th, 1 ms 95th, 2 ms 99th, 10 ms 99.9th.
   ```
   I configured ` --throughput -1` to disable throttling, we can see, the tps is about `897,311`
   * Consumer side
   ```
   # Fetch 5,000,000  record (running for about 3 seconds)
   /opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 5000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
   start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
   2021-01-28 11:46:58:466, 2021-01-28 11:47:01:299, 476.8650, 168.3251, 5000292, 1765016.5902, 1611802018934, -1611802016101, -0.0000, -0.0031
   
   # Fetch 10,000,000   records  (running for about 5 seconds)
   /opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 10000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
   start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
   2021-01-28 11:45:58:096, 2021-01-28 11:46:03:224, 953.6926, 185.9775, 10000192, 1950115.4446, 1611801958563, -1611801953435, -0.0000, -0.0062
   
   # Fetch 1,000,000,000 records (running for about 20 minutes)
   /opt/kafka_2.13-2.7.0/bin/kafka-consumer-perf-test.sh  --topic scheduler0 --messages 1000000000 --bootstrap-server=xxx.xxx.xxx.xxx:9093,xxx.xxx.xxx.xxx:9094,xxx.xxx.xxx.xxx:9095
   start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
   2021-01-28 11:55:33:186, 2021-01-28 12:16:03:855, 95367.4615, 77.4924, 1000000313, 812566.4277, 1611802533652, -1611801302983, -0.0001, -0.6204
   ```
   We can saw the the tps is very high as well, when fetch `5,000,000`, the tps is `1,765,016`, when fetch `10,000,000` the tps is `1,950,115`, when fetch `1,000,000,000`, the tps is `81,2566`
   
   I also tested below scenes
   when running kafka-producer-perf-test.sh, i adjusted the `--num-records` to huge value to make the producer script running for long time, when running kafka-consumer-perf-test.sh, i ajusted the `--messages` to huge value as well to make the consumer script running for long time
   
   During benchmarking, i observed the cpu/memory doesn't consume too much for all kafka nodes
   
   
   FYI, on our production env, the consumer.commit offset timeout(>=5.seconds) issue didn't reappeared after we used the latest kafka version.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] style95 commented on pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
style95 commented on pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046#issuecomment-762064350


   Just for those who visit this PR, [that issue](https://issues.apache.org/jira/browse/KAFKA-8334) makes committing offests take more than 5 seconds, accordingly it would dramatically increase the wait time of certain activations.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [openwhisk] ningyougang closed pull request #5046: Make kafka version configurable

Posted by GitBox <gi...@apache.org>.
ningyougang closed pull request #5046:
URL: https://github.com/apache/openwhisk/pull/5046


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org