You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/20 00:48:10 UTC

[GitHub] [pulsar] liangyepianzhou opened a new pull request, #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

liangyepianzhou opened a new pull request, #16917:
URL: https://github.com/apache/pulsar/pull/16917

   Master Issue: https://github.com/apache/pulsar/issues/16913
   ### Motivation
   1. Add configurations.
   2. Implement system topic client for snapshot topic and index topic.
   3. Implement AbortedTxnProcessor.
   4. Add transaction buffer snapshot metrics.
   ### Modification
   Add configuration
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   *(Please pick either of the following options)*
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This change is already covered by existing tests, such as *(please describe tests)*.
   
   *(or)*
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - *Added integration tests for end-to-end deployment with large payloads (10MB)*
     - *Extended integration test for recovery after broker failure*
   
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
     - Dependencies (does it add or upgrade a dependency): (yes / no)
     - The public API: (yes / no)
     - The schema: (yes / no / don't know)
     - The default values of configurations: (yes / no)
     - The wire protocol: (yes / no)
     - The rest endpoints: (yes / no)
     - The admin cli options: (yes / no)
     - Anything that affects deployment: (yes / no / don't know)
   
   ### Documentation
   
   Check the box below or label this PR directly.
   
   Need to update docs? 
   
   - [ ] `doc-required` 
   (Your PR needs to update docs and you will update later)
     
   - [x] `doc-not-needed` 
   (Please explain why)
     
   - [ ] `doc` 
   (Your PR contains doc changes)
   
   - [ ] `doc-complete`
   (Docs have been already added)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on pull request #16917: [Improve][Txn]TransactionBuffer multiple-snapshot configuration

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#issuecomment-1236026893

   The pr had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] congbobo184 commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r969082782


##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##########
@@ -2731,15 +2731,34 @@ public class ServiceConfiguration implements PulsarConfiguration {
     @FieldContext(
             category = CATEGORY_TRANSACTION,
             doc = "Transaction buffer take snapshot transaction count"
+                    + "If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata"
+                    + "after the number of transaction operations reaches this value."
     )
     private int transactionBufferSnapshotMaxTransactionCount = 1000;
 
     @FieldContext(
             category = CATEGORY_TRANSACTION,
-            doc = "Transaction buffer take snapshot min interval time"
+            doc = "The interval time for transaction buffer to take snapshots."
+                    + "If transaction buffer enables snapshot segment, "
+                    + "it is the interval time for transaction buffer to update snapshot metadata."
     )
     private int transactionBufferSnapshotMinTimeInMillis = 5000;
 
+    @FieldContext(
+            category = CATEGORY_SERVER,
+            doc = "Transaction buffer stores the transaction ID of aborted transactions and takes snapshots."
+                    + "This configuration determines the size of the snapshot segment. "
+                    + "The default value is 256 KB (262144 bytes)."
+    )
+    private int transactionBufferSnapshotSegmentSize = 262144;
+
+    @FieldContext(
+            category = CATEGORY_SERVER,

Review Comment:
   ```suggestion
               category = CATEGORY_TRANSACTION,
   ```



##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java:
##########
@@ -2731,15 +2731,34 @@ public class ServiceConfiguration implements PulsarConfiguration {
     @FieldContext(
             category = CATEGORY_TRANSACTION,
             doc = "Transaction buffer take snapshot transaction count"
+                    + "If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata"
+                    + "after the number of transaction operations reaches this value."
     )
     private int transactionBufferSnapshotMaxTransactionCount = 1000;
 
     @FieldContext(
             category = CATEGORY_TRANSACTION,
-            doc = "Transaction buffer take snapshot min interval time"
+            doc = "The interval time for transaction buffer to take snapshots."
+                    + "If transaction buffer enables snapshot segment, "
+                    + "it is the interval time for transaction buffer to update snapshot metadata."
     )
     private int transactionBufferSnapshotMinTimeInMillis = 5000;
 
+    @FieldContext(
+            category = CATEGORY_SERVER,

Review Comment:
   ```suggestion
               category = CATEGORY_TRANSACTION,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r963317014


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.
+# If transaction buffer enables snapshot segment, this is transaction updating snapshot metadata internal time.
 # Unit : millisecond
 transactionBufferSnapshotMinTimeInMillis=5000
 
+# Whether to enable segmented transaction buffer snapshot which is able to handle a large number of aborted transactions.
+transactionBufferSegmentedSnapshotEnabled = false
+
+# Transaction buffer will stored the transaction ID of the aborted transaction, and take snapshot for it.
+# This configuration is configured to determine the size of the snapshot segment. defualt:256kb.

Review Comment:
   ```suggestion
   # This configuration determines the size of the snapshot segment. The default value is 256 KB (262144 bytes).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] congbobo184 commented on pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
congbobo184 commented on PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#issuecomment-1244820512

   when https://lists.apache.org/thread/bqoy3oz8flvxy7xpmnw81cr4c9sz5vy0 +3 binding can merge


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui closed pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
codelipenghui closed pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration
URL: https://github.com/apache/pulsar/pull/16917


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] liangyepianzhou commented on pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#issuecomment-1246118041

   > when https://lists.apache.org/thread/bqoy3oz8flvxy7xpmnw81cr4c9sz5vy0 +3 binding can merge
   
   It already has +3 binding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] liangyepianzhou commented on pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#issuecomment-1237655009

   @codelipenghui @congbobo184 @gaoran10 Please help review this PR when you have time. THX.
   @momo-jun Please help review this PR from a technical writer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r963312345


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.
+# If transaction buffer enables snapshot segment, this is transaction updating snapshot metadata internal time.

Review Comment:
   ```suggestion
   # If transaction buffer enables snapshot segment, it is the interval time for transaction buffer to update snapshot metadata.
   ```



##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.

Review Comment:
   ```suggestion
   # The interval time for transaction buffer to take snapshots.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r974171995


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# The interval time for transaction buffer to take snapshots.
+# If transaction buffer enables snapshot segment, it is the interval time for transaction buffer to update snapshot metadata.
 # Unit : millisecond
 transactionBufferSnapshotMinTimeInMillis=5000
 
+# Whether to enable segmented transaction buffer snapshot to handle a large number of aborted transactions.
+transactionBufferSegmentedSnapshotEnabled = false

Review Comment:
   ```suggestion
   transactionBufferSegmentedSnapshotEnabled=false
   ```



##########
conf/standalone.conf:
##########
@@ -1108,12 +1108,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# The interval time for transaction buffer to take snapshots.
+# If transaction buffer enables snapshot segment, it is the interval time for transaction buffer to update snapshot metadata.
 # Unit : millisecond
 transactionBufferSnapshotMinTimeInMillis=5000
 
+# Whether to enable segmented transaction buffer snapshot to handle a large number of aborted transactions.
+transactionBufferSegmentedSnapshotEnabled = false

Review Comment:
   ```suggestion
   transactionBufferSegmentedSnapshotEnabled=false
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r963314669


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.
+# If transaction buffer enables snapshot segment, this is transaction updating snapshot metadata internal time.
 # Unit : millisecond
 transactionBufferSnapshotMinTimeInMillis=5000
 
+# Whether to enable segmented transaction buffer snapshot which is able to handle a large number of aborted transactions.
+transactionBufferSegmentedSnapshotEnabled = false
+
+# Transaction buffer will stored the transaction ID of the aborted transaction, and take snapshot for it.

Review Comment:
   ```suggestion
   # Transaction buffer stores the transaction ID of aborted transactions and takes snapshots.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r963312345


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.
+# If transaction buffer enables snapshot segment, this is transaction updating snapshot metadata internal time.

Review Comment:
   ```suggestion
   # If the transaction buffer enables snapshot segment, it is the interval time for the transaction buffer to update snapshot metadata.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] liangyepianzhou merged pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
liangyepianzhou merged PR #16917:
URL: https://github.com/apache/pulsar/pull/16917


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#issuecomment-1251712532

   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r963312916


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.
+# If transaction buffer enables snapshot segment, this is transaction updating snapshot metadata internal time.
 # Unit : millisecond
 transactionBufferSnapshotMinTimeInMillis=5000
 
+# Whether to enable segmented transaction buffer snapshot which is able to handle a large number of aborted transactions.

Review Comment:
   ```suggestion
   # Whether to enable segmented transaction buffer snapshot to handle a large number of aborted transactions.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] momo-jun commented on a diff in pull request #16917: [improve][txn][PIP-196]Segmented transaction buffer snapshot configuration

Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #16917:
URL: https://github.com/apache/pulsar/pull/16917#discussion_r963311516


##########
conf/broker.conf:
##########
@@ -1497,12 +1497,22 @@ transactionCoordinatorEnabled=false
 transactionMetadataStoreProviderClassName=org.apache.pulsar.transaction.coordinator.impl.MLTransactionMetadataStoreProvider
 
 # Transaction buffer takes a snapshot after the number of transaction operations reaches this value.
+# If transaction buffer enables snapshot segment, transaction buffer updates snapshot metadata
+# after the number of transaction operations reaches this value.
 transactionBufferSnapshotMaxTransactionCount=1000
 
-# Transaction buffer take snapshot interval time
+# Transaction buffer take snapshot interval time.

Review Comment:
   ```suggestion
   # The interval time for the transaction buffer to take snapshots.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org