You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/04/15 21:24:59 UTC
[GitHub] [pulsar] klwilson227 opened a new pull request #10246: Add support for deduplication testing
klwilson227 opened a new pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246
Add some basic support for testing deduplication feature.
1) Ability to set the producer name in the test client.
2) Ability to set the sendTimeout in the test client.
3) Ability to set the separator used between producername and threadnumber. As well as the topic and topic number. Defaults to '-'
4) Ability to customize format each payload record if enabled.
- Ability to customize/replace the formatting class which is used.
- DefaultFormatter provided:
- MetaData replacements: %p replaced with producer Name, %t nanoseconds when replacement occured, %i external messageId
- Other replacements follow java sprintf formatting with %d %s %f %l with randomly generated strings or numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat merged pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#issuecomment-826615027
This PR introduced a flaky test, reported as #10378. I have created a PR to fix the flakiness in #10379 . Please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] klwilson227 commented on pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
klwilson227 commented on pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#issuecomment-821209966
Merlimat, RE the consumer side, I'm sure the consumer side would also benefit as well for a possible demo/software acceptance level validation. However, in my particular use case the validation and scanning for duplicates and lost records was easier implemented in the Data Sink which received the data hence I did not invest extra to build a data validation layer within pulsar test client, only the ability to publish data using both the dedup enabled and dedup disabled.
Since, I did not have the requirement, it was not coded. And I did not want to assume that this was a requirement, as a poor implementation may show both strengths and weaknesses in the platform. I can certainly take it in that direction. But would think that may be a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] klwilson227 commented on a change in pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
klwilson227 commented on a change in pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#discussion_r614486173
##########
File path: pulsar-testclient/pom.xml
##########
@@ -94,6 +94,13 @@
<groupId>org.hdrhistogram</groupId>
<artifactId>HdrHistogram</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.json</groupId>
+ <artifactId>json</artifactId>
+ <version>20201115</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
Thanks for the feedback, I will take a look at updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] klwilson227 commented on a change in pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
klwilson227 commented on a change in pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#discussion_r614861660
##########
File path: pulsar-testclient/pom.xml
##########
@@ -94,6 +94,13 @@
<groupId>org.hdrhistogram</groupId>
<artifactId>HdrHistogram</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.json</groupId>
+ <artifactId>json</artifactId>
+ <version>20201115</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
Move completed to jackson.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on a change in pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#discussion_r614472569
##########
File path: pulsar-testclient/pom.xml
##########
@@ -94,6 +94,13 @@
<groupId>org.hdrhistogram</groupId>
<artifactId>HdrHistogram</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.json</groupId>
+ <artifactId>json</artifactId>
+ <version>20201115</version>
+ <scope>test</scope>
+ </dependency>
Review comment:
The license of this library is `The Software shall be used for Good, not Evil.` and it's not really compatible with Apache license. We generally use Jackson and it's already in class path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] klwilson227 commented on pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
klwilson227 commented on pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#issuecomment-823724669
/pulsar of rerun CI-CPP,Python Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] klwilson227 edited a comment on pull request #10246: Add support for deduplication testing
Posted by GitBox <gi...@apache.org>.
klwilson227 edited a comment on pull request #10246:
URL: https://github.com/apache/pulsar/pull/10246#issuecomment-823724669
/pulsabot rerun CI-CPP,Python Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org