You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "rreddy-22 (via GitHub)" <gi...@apache.org> on 2025/06/06 00:10:21 UTC
[PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
rreddy-22 opened a new pull request, #19910:
URL: https://github.com/apache/kafka/pull/19910
When InitProducerId is handled on the transaction coordinator, the producer epoch is incremented (so that we fence stale requests), then if a transaction was ongoing during this time, it's aborted. With transaction version 2 (a.k.a. KIP-890 part 2), abort increments the producer epoch again (it's the part of the new abort / commit protocol), so the epoch ends up incremented twice.
In most cases, this is benign, but in the case where the epoch of the ongoing transaction is 32766, it's incremented to 32767, which is the maximum value for short. Then, when it's incremented for the second time, it goes negative, causing an illegal argument exception.
To fix this we just avoid bumping the epoch a second time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2134959821
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
Review Comment:
oh yes! Sorry I did this one in a hurry so I might've missed some things!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "artemlivshits (via GitHub)" <gi...@apache.org>.
artemlivshits commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2131102531
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -828,7 +828,16 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
nextProducerIdOrErrors.flatMap {
nextProducerId =>
- Right(coordinatorEpoch, txnMetadata.prepareAbortOrCommit(nextState, clientTransactionVersion, nextProducerId.asInstanceOf[Long], time.milliseconds(), noPartitionAdded))
+ // For epoch fence case, we don't want to bump the epoch again in prepareAbortOrCommit
+ // since it was already bumped during the fencing phase to avoid double increment
+ val transactionVersionForCall = if (isEpochFence) {
+ // Use TV_0 to prevent epoch bump since it was already bumped during fencing
+ TransactionVersion.TV_0
Review Comment:
Instead of doing this, let's just do something at line 825 -- the TV will move forward with KIP-939 and using old version could lead to subtle bugs over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133075311
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
Review Comment:
Wouldn't this path be covered by
```
if (!clientTransactionVersion.supportsEpochBump()) {
endTransactionWithTV1(transactionalId, producerId, producerEpoch, txnMarkerResult, isFromClient, responseCallback, requestLocal)
return
}
```
earlier in the method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133073869
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
Review Comment:
If I understand correctly, the goal is to keep this logic for TV1 but not use it for TV2. If so lgtm 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133074219
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
Review Comment:
Should this comment be changed back to reflect TV1 vs TV2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "artemlivshits (via GitHub)" <gi...@apache.org>.
artemlivshits commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133128442
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
+ // For TV1, manually bump // the epoch in the transaction metadata we are about to append.
+ txnMetadata.producerEpoch = producerEpoch
+ txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
Review Comment:
Could you check whether we need to reset lastProducerEpoch even for TV2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on PR #19910:
URL: https://github.com/apache/kafka/pull/19910#issuecomment-2967520528
Will pick to 4.0 and 4.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133075311
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
Review Comment:
Wouldn't this path be covered by
``` if (!clientTransactionVersion.supportsEpochBump()) {
endTransactionWithTV1(transactionalId, producerId, producerEpoch, txnMarkerResult, isFromClient, responseCallback, requestLocal)
return
}
```
earlier in the method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2132508413
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -828,7 +828,16 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
nextProducerIdOrErrors.flatMap {
nextProducerId =>
- Right(coordinatorEpoch, txnMetadata.prepareAbortOrCommit(nextState, clientTransactionVersion, nextProducerId.asInstanceOf[Long], time.milliseconds(), noPartitionAdded))
+ // For epoch fence case, we don't want to bump the epoch again in prepareAbortOrCommit
+ // since it was already bumped during the fencing phase to avoid double increment
+ val transactionVersionForCall = if (isEpochFence) {
+ // Use TV_0 to prevent epoch bump since it was already bumped during fencing
+ TransactionVersion.TV_0
Review Comment:
yess I was also concerned this was a little hacky solution that might cause side effects, let me find another way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan merged PR #19910:
URL: https://github.com/apache/kafka/pull/19910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2136822173
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
+ // For TV1, manually bump // the epoch in the transaction metadata we are about to append.
+ txnMetadata.producerEpoch = producerEpoch
+ txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
Review Comment:
Yeah if I recall correctly, for tv2, I wanted to try to avoid setting fields outside of the actual transition. It was just this weird epoch fence state that does things differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "rreddy-22 (via GitHub)" <gi...@apache.org>.
rreddy-22 commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2136535090
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
+ // For TV1, manually bump // the epoch in the transaction metadata we are about to append.
+ txnMetadata.producerEpoch = producerEpoch
+ txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
Review Comment:
For TV2, it looks like the last producer epoch is immediately set in prepareAbortOrCommit, so we don't need to set it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133074219
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
Review Comment:
Should this comment be changed back to reflect TV1 vs TV2?
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
Review Comment:
So we can just remove this code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2133073869
##########
core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala:
##########
@@ -819,11 +819,14 @@ class TransactionCoordinator(txnConfig: TransactionConfig,
}
if (nextState == TransactionState.PREPARE_ABORT && isEpochFence) {
- // We should clear the pending state to make way for the transition to PrepareAbort and also bump
- // the epoch in the transaction metadata we are about to append.
+ // We should clear the pending state to make way for the transition to PrepareAbort
txnMetadata.pendingState = None
- txnMetadata.producerEpoch = producerEpoch
- txnMetadata.lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH
+ if (!clientTransactionVersion.supportsEpochBump()) {
Review Comment:
If I understand correctly, the goal is to keep this logic for TV1 but not use it for TV2. If so lgtm 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19367: Fix InitProducerId with TV2 double-increments epoch if ongoing transaction is aborted [kafka]
Posted by "jolshan (via GitHub)" <gi...@apache.org>.
jolshan commented on code in PR #19910:
URL: https://github.com/apache/kafka/pull/19910#discussion_r2141000080
##########
core/src/test/scala/unit/kafka/coordinator/transaction/TransactionCoordinatorTest.scala:
##########
@@ -1327,6 +1327,72 @@ class TransactionCoordinatorTest {
any())
}
+ @Test
+ def shouldHandleTimeoutAtEpochOverflowBoundaryCorrectlyTV2(): Unit = {
+ // Test the scenario where we have an ongoing transaction at epoch 32766 (Short.MaxValue - 1)
+ // and the producer crashes/times out. This test verifies that the timeout handling
+ // correctly manages the epoch overflow scenario without causing failures.
+
+ val epochAtMaxBoundary = (Short.MaxValue - 1).toShort // 32766
+ val now = time.milliseconds()
+
+ // Create transaction metadata at the epoch boundary that would cause overflow if double-incremented
+ val txnMetadata = new TransactionMetadata(
+ transactionalId = transactionalId,
+ producerId = producerId,
+ prevProducerId = RecordBatch.NO_PRODUCER_ID,
+ nextProducerId = RecordBatch.NO_PRODUCER_ID,
+ producerEpoch = epochAtMaxBoundary,
+ lastProducerEpoch = RecordBatch.NO_PRODUCER_EPOCH,
+ txnTimeoutMs = txnTimeoutMs,
+ state = TransactionState.ONGOING,
+ topicPartitions = partitions,
+ txnStartTimestamp = now,
+ txnLastUpdateTimestamp = now,
+ clientTransactionVersion = TV_2
+ )
+ assertTrue(txnMetadata.isProducerEpochExhausted)
+
+ // Mock the transaction manager to return our test transaction as timed out
+ when(transactionManager.timedOutTransactions())
+ .thenReturn(List(TransactionalIdAndProducerIdEpoch(transactionalId, producerId, epochAtMaxBoundary)))
+ when(transactionManager.getTransactionState(ArgumentMatchers.eq(transactionalId)))
+ .thenReturn(Right(Some(CoordinatorEpochAndTxnMetadata(coordinatorEpoch, txnMetadata))))
+ when(transactionManager.transactionVersionLevel()).thenReturn(TV_2)
+
+ // Mock the append operation to simulate successful write
+ when(transactionManager.appendTransactionToLog(
+ ArgumentMatchers.eq(transactionalId),
+ ArgumentMatchers.eq(coordinatorEpoch),
+ any[TxnTransitMetadata],
+ capturedErrorsCallback.capture(),
+ any(),
+ any())
+ ).thenAnswer(_ => capturedErrorsCallback.getValue.apply(Errors.NONE))
+
+ // Track the actual behavior
+ var callbackInvoked = false
+ var resultError: Errors = null
+ var resultProducerId: Long = -1
+ var resultEpoch: Short = -1
+
+ def checkOnEndTransactionComplete(txnIdAndPidEpoch: TransactionalIdAndProducerIdEpoch)
+ (error: Errors, newProducerId: Long, newProducerEpoch: Short): Unit = {
+ callbackInvoked = true
+ resultError = error
+ resultProducerId = newProducerId
+ resultEpoch = newProducerEpoch
+ println(s"Timeout callback invoked with error: $error, producerId: $newProducerId, epoch: $newProducerEpoch")
Review Comment:
Did we want to verify the callback is invoked and that the values are as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org