You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/08/18 03:04:03 UTC

[GitHub] [pulsar] gaoran10 opened a new pull request #7833: [Transaction] Consume transaction message logic optimized optimized

gaoran10 opened a new pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833


   ### Motivation
   
   The transaction message consuming logic needs to be optimized. One transaction commit marker in the topic partition consists of many 
   
   ### Modifications
   
   1. When send 
   
   ### Verifying this change
   
   This change can be verified as follows:
   
   - *org.apache.pulsar.broker.transaction.TransactionConsumeTest*
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API: (no)
     - The schema: (no)
     - The default values of configurations: (no)
     - The wire protocol: (no)
     - The rest endpoints: (no)
     - The admin cli options: (no)
     - Anything that affects deployment: (no)
   
   ### Documentation
   
     - Does this pull request introduce a new feature? (no)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a change in pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#discussion_r478549595



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
##########
@@ -2857,5 +2857,14 @@ private int applyMaxSizeCap(int maxEntries, long maxSizeBytes) {
 
         return Math.min(maxEntriesBasedOnSize, maxEntries);
     }
+
+    public void internalInitBatchDeletedIndex(PositionImpl position, int totalNumMessageInBatch) {
+        if (!batchDeletedIndexes.containsKey(position)) {
+            BitSetRecyclable bitSetRecyclable = BitSetRecyclable.create();
+            bitSetRecyclable.set(0, totalNumMessageInBatch);
+            batchDeletedIndexes.put(position, bitSetRecyclable);
+        }

Review comment:
       I'll use this method instead.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui merged pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#issuecomment-682262883


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#issuecomment-678019791


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a change in pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#discussion_r478382478



##########
File path: .github/workflows/ci-unit-broker-broker-gp1.yaml
##########
@@ -65,13 +65,13 @@ jobs:
         if: steps.docs.outputs.changed_only == 'no'
         run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='**/AdminApiOffloadTest.java' -DtestForkCount=1 -DtestReuseFork=true
 
-      - name: run flaky test '**/TransactionProduceTest.java'
+      - name: run transaction test
         if: steps.docs.outputs.changed_only == 'no'
-        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='**/TransactionProduceTest.java' -DtestForkCount=1 -DtestReuseFork=true
+        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/transaction/**/*.java,org/apache/pulsar/client/transaction/**/*.java' -DfailIfNoTests=false -DtestForkCount=1 -DtestReuseFork=true
 
       - name: run unit tests for pulsar-broker
         if: steps.docs.outputs.changed_only == 'no'
-        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/**/*.java' -Dexclude='org/apache/pulsar/broker/zookeeper/**/*.java,org/apache/pulsar/broker/loadbalance/**/*.java,org/apache/pulsar/broker/service/**/*.java,**/AdminApiOffloadTest.java,**/TransactionProduceTest.java'
+        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/**/*.java' -Dexclude='org/apache/pulsar/broker/zookeeper/**/*.java,org/apache/pulsar/broker/loadbalance/**/*.java,org/apache/pulsar/broker/service/**/*.java,**/AdminApiOffloadTest.java,org/apache/pulsar/broker/transaction/**/*.java,org/apache/pulsar/client/transaction/**/*.java'

Review comment:
       I want to sperate transaction tests to make the transaction tests are executed in serial.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a change in pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#discussion_r478232637



##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedCursorImpl.java
##########
@@ -2857,5 +2857,14 @@ private int applyMaxSizeCap(int maxEntries, long maxSizeBytes) {
 
         return Math.min(maxEntriesBasedOnSize, maxEntries);
     }
+
+    public void internalInitBatchDeletedIndex(PositionImpl position, int totalNumMessageInBatch) {
+        if (!batchDeletedIndexes.containsKey(position)) {
+            BitSetRecyclable bitSetRecyclable = BitSetRecyclable.create();
+            bitSetRecyclable.set(0, totalNumMessageInBatch);
+            batchDeletedIndexes.put(position, bitSetRecyclable);
+        }

Review comment:
       batchDeletedIndexes.computeIfAbsent().

##########
File path: .github/workflows/ci-unit-broker-broker-gp1.yaml
##########
@@ -65,13 +65,13 @@ jobs:
         if: steps.docs.outputs.changed_only == 'no'
         run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='**/AdminApiOffloadTest.java' -DtestForkCount=1 -DtestReuseFork=true
 
-      - name: run flaky test '**/TransactionProduceTest.java'
+      - name: run transaction test
         if: steps.docs.outputs.changed_only == 'no'
-        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='**/TransactionProduceTest.java' -DtestForkCount=1 -DtestReuseFork=true
+        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/transaction/**/*.java,org/apache/pulsar/client/transaction/**/*.java' -DfailIfNoTests=false -DtestForkCount=1 -DtestReuseFork=true
 
       - name: run unit tests for pulsar-broker
         if: steps.docs.outputs.changed_only == 'no'
-        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/**/*.java' -Dexclude='org/apache/pulsar/broker/zookeeper/**/*.java,org/apache/pulsar/broker/loadbalance/**/*.java,org/apache/pulsar/broker/service/**/*.java,**/AdminApiOffloadTest.java,**/TransactionProduceTest.java'
+        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/**/*.java' -Dexclude='org/apache/pulsar/broker/zookeeper/**/*.java,org/apache/pulsar/broker/loadbalance/**/*.java,org/apache/pulsar/broker/service/**/*.java,**/AdminApiOffloadTest.java,org/apache/pulsar/broker/transaction/**/*.java,org/apache/pulsar/client/transaction/**/*.java'

Review comment:
       Shall we need this change? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#issuecomment-675902856


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#issuecomment-678761654


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on a change in pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on a change in pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#discussion_r478382478



##########
File path: .github/workflows/ci-unit-broker-broker-gp1.yaml
##########
@@ -65,13 +65,13 @@ jobs:
         if: steps.docs.outputs.changed_only == 'no'
         run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='**/AdminApiOffloadTest.java' -DtestForkCount=1 -DtestReuseFork=true
 
-      - name: run flaky test '**/TransactionProduceTest.java'
+      - name: run transaction test
         if: steps.docs.outputs.changed_only == 'no'
-        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='**/TransactionProduceTest.java' -DtestForkCount=1 -DtestReuseFork=true
+        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/transaction/**/*.java,org/apache/pulsar/client/transaction/**/*.java' -DfailIfNoTests=false -DtestForkCount=1 -DtestReuseFork=true
 
       - name: run unit tests for pulsar-broker
         if: steps.docs.outputs.changed_only == 'no'
-        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/**/*.java' -Dexclude='org/apache/pulsar/broker/zookeeper/**/*.java,org/apache/pulsar/broker/loadbalance/**/*.java,org/apache/pulsar/broker/service/**/*.java,**/AdminApiOffloadTest.java,**/TransactionProduceTest.java'
+        run: ./build/retry.sh mvn -B -ntp test -pl pulsar-broker -Dinclude='org/apache/pulsar/broker/**/*.java' -Dexclude='org/apache/pulsar/broker/zookeeper/**/*.java,org/apache/pulsar/broker/loadbalance/**/*.java,org/apache/pulsar/broker/service/**/*.java,**/AdminApiOffloadTest.java,org/apache/pulsar/broker/transaction/**/*.java,org/apache/pulsar/client/transaction/**/*.java'

Review comment:
       I want to sperate transaction tests to make the transaction tests are executed in serial.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#issuecomment-682058263


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] gaoran10 commented on pull request #7833: [Transaction] Consume transaction message logic optimized

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #7833:
URL: https://github.com/apache/pulsar/pull/7833#issuecomment-678764379


   /pulsarbot run-failure-checks


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org