You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "thetumbled (via GitHub)" <gi...@apache.org> on 2023/03/10 07:07:40 UTC

[GitHub] [pulsar] thetumbled opened a new pull request, #19781: [feat][cli] Support for transaction consistence test with PerformanceProducer.

thetumbled opened a new pull request, #19781:
URL: https://github.com/apache/pulsar/pull/19781

   <!-- Either this PR fixes an issue, -->
   
   Fixes #19780
   
   
   ### Motivation
   enhance `PerformanceProducer` to support for transaction consistence test.
   
   ### Modifications
   - implement a message formatter to generate specific message format.
   - implement client-side persistence scheme.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/thetumbled/pulsar/pull/18
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat][cli] Support for transaction consistence test with PerformanceProducer. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #19781:
URL: https://github.com/apache/pulsar/pull/19781#issuecomment-6182316432

   Hi @thetumbled, thanks for the PR, and sorry it never got a proper review. PerformanceProducer has since been split into PerformanceProducerBase plus separate V4 and V5 implementations, so this change no longer applies to current master and would need to be redone. We're going through the backlog of open PRs (https://lists.apache.org/thread/cgy7w0qgnq8s7j0ykyzgrn3hj4nb67wf), so I'm closing this one. If you'd like to continue, feel free to open a new PR against the current perf tool.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] thetumbled commented on pull request #19781: [feat][cli] Support for transaction consistence test with PerformanceProducer.

Posted by "thetumbled (via GitHub)" <gi...@apache.org>.
thetumbled commented on PR #19781:
URL: https://github.com/apache/pulsar/pull/19781#issuecomment-1463402425

   BTW, this PR do not contain the code for reading content from topics and check for the data integrity, because i implement it out of pulsar project. the detailed code is as follows:
   ```
   package org.example.utils;
   
   public class PulsarCommonUtils {
       private static final Logger log = LoggerFactory.getLogger(PulsarCommonUtils.class);
   
       @Parameter(names = "-topic", description = "topic name")
       String topic = "persistent://test/test/testTxn10";
   
       @Parameter(names = "-sub", description = "subscription name")
       String sub = "sub";
   
       @Parameter(names = { "-u", "--service-url" }, description = "Pulsar Service URL")
       public String serviceURL;
   
       @Parameter(names = "-token", description ="token used to authenticate.")
       public String token = "";
   
       @Parameter(names = "--help", description = "help info", help = true)
       private boolean help = false;
   
   
       // check the content of topic whether constitute a continuous range
       public static boolean checkCorrectness(PulsarCommonUtils pulsarCommonUtils) throws PulsarClientException {
           boolean result = true;
           if (pulsarCommonUtils.serviceURL == null || pulsarCommonUtils.serviceURL.length() == 0) {
               log.error("please input Pulsar Service URL");
               return false;
           }
           PulsarClient client = PulsarClient.builder()
                   .serviceUrl(pulsarCommonUtils.serviceURL)
                   .authentication(AuthenticationFactory.token(pulsarCommonUtils.token))
                   .build();
           Consumer<byte[]> consumer = client.newConsumer()
                   .topic(pulsarCommonUtils.topic)
                   .subscriptionName(pulsarCommonUtils.sub)
                   .subscribe();
           RangeSet<Long> rangeSet = TreeRangeSet.create();
           Message msg;
           Long id;
           long msgCount = 0l;
           long duplicateCount = 0l;
           while (true) {
               msg = consumer.receive(10, TimeUnit.SECONDS);
               if (msg == null) {
                   break;
               }
               msgCount++;
               id = CommonUtils.bytesToLong(msg.getData());
               if (rangeSet.contains(id)) {
                   duplicateCount++;
               } else {
                   // for n, we add range [n,n+1) into rangeSet.
                   rangeSet.add(Range.closedOpen(id, id + 1));
               }
               consumer.acknowledge(msg);
           }
           // check for message loss
           Set<Range<Long>> set = rangeSet.asRanges();
           if (set.size() > 1) {
               long validCount = 0;
               for (Range<Long> range : set) {
                   validCount += range.upperEndpoint() - range.lowerEndpoint();
               }
               log.info("Some messages are missing! more than one range exists." +
                       "Valid messages Count:{} RangeSet:{}", validCount, rangeSet);
               result = false;
           }else if (set.size() == 0) {
               log.info("there is no any message!");
               return true;
           }else{
               log.info("only one range exists. RangeSet:{}", rangeSet);
           }
   
           // check for message duplication
           if (duplicateCount != 0) {
               result = false;
           }
           log.info("duplicate message count:{}, total message count:{}, duplicate rate:{}%",
                   duplicateCount, msgCount, 100.0 * duplicateCount / msgCount);
   
   
           client.close();
           return result;
       }
   
       public static void main(String[] args) {
           PulsarCommonUtils pulsarCommonUtils = new PulsarCommonUtils();
           JCommander jcommander = JCommander.newBuilder().addObject(pulsarCommonUtils).build();
           try {
               jcommander.parse(args);
               if (pulsarCommonUtils.help) {
                   jcommander.usage();
                   System.exit(1);
               } else {
                   if (pulsarCommonUtils.checkCorrectness) {
                       checkCorrectness(pulsarCommonUtils);
                   } 
               }
           } catch (Exception e) {
               System.err.println(e);
               System.exit(1);
           }
       }
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on pull request #19781: [feat][cli] Support for transaction consistence test with PerformanceProducer.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19781:
URL: https://github.com/apache/pulsar/pull/19781#issuecomment-1507816576

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat][cli] Support for transaction consistence test with PerformanceProducer. [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed pull request #19781: [feat][cli] Support for transaction consistence test with PerformanceProducer.
URL: https://github.com/apache/pulsar/pull/19781


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org