You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/12/02 13:23:44 UTC

[GitHub] [pulsar-client-go] bschofield opened a new pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

bschofield opened a new pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678


   Fixes issue https://github.com/apache/pulsar-client-go/issues/513.
   
   This issue was previously partially addressed by @zzzming in PR https://github.com/apache/pulsar-client-go/pull/528.
   
   ### Motivation
   
   Following PR https://github.com/apache/pulsar-client-go/pull/528, the logic for the `hasSpace()` function, which is used to determine whether a batch container has enough space for a new message, reads:
   
   ```
   func (bc *keyBasedBatchContainer) hasSpace(payload []byte) bool {
   	msgSize := uint32(len(payload))
   	return bc.numMessages+1 < bc.maxMessages || (bc.buffer.ReadableBytes()+msgSize) < uint32(bc.maxBatchSize)
   }
   ```
   
   Currently, then, the batch container will be considered to have space (be non-full) until **both** the max-messages and the max-size limits are hit.
   
   I think that is incorrect. Nearby in the code, the batch container is considered to be full if **either** of the conditions are hit:
   
   ```
   // IsFull check if the size in the current batch exceeds the maximum size allowed by the batch
   func (bc *batchContainer) IsFull() bool {
   	return bc.numMessages >= bc.maxMessages || bc.buffer.ReadableBytes() > uint32(bc.maxBatchSize)
   }
   ```
   
   Because the logical rule is `!(A || B) = (!A) && (!B)`, the `||` in `hasSpace()` should be an `&&`.
   
   This issue was causing me trouble because I set `maxMessages = math.MaxUint32`, with the intention that batching be driven only by message size. With the current logic, that meant that `hasSpace()` always returned true, so I was creating very large message batches which caused system problems.
   
   ### Modifications
   
   Change the `||` in `hasSpace()` to an `&&`, so that the container will be considered full if either limit is hit.
   
   ### Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ### Documentation
   
     - Does this pull request introduce a new feature? no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield commented on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-984722305


   The tests now seem to be failing on `TestNacksTracker()`. Is that a consequence of this change, or an unrelated issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and `(*negativeAcksTracker).track()` also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions. Also the documentation says that `time.Ticker` is allowed to drop ticks, which I guess could be happening. A longer sleep might be a reasonable workaround.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] cckellogg commented on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
cckellogg commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985073999


   > The tests now seem to be failing on `TestNacksTracker()`. Is that a consequence of this change, or an unrelated issue?
   
   It's probably unrelated since the tests passed for the over go versions. The concerning part is there may be a flaky `TestNacksTracker()` test now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   // since the client ticks at an interval of delay / 3
   // wait another interval to ensure we get all messages
   time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and `(*negativeAcksTracker).track()` also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions. Also the documentation says that `time.Ticker` is allowed to drop ticks, which I suppose could be happening.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and `(*negativeAcksTracker).track()` also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] wolfstudy merged pull request #678: [Issue 513] Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
wolfstudy merged pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and `(*negativeAcksTracker).track()` also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions. Also the documentation says that `time.Ticker` is allowed to drop ticks, which I guess could be happening.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield commented on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985394760


   I added `TestMaxBatchSize()` to `producer_test.go`, based on `TestMaxBatchSize()`.
   
   From the results of that test, it looks like the `<` in the size test should actually be `<=`, to accommodate the case where the new payload fits exactly in the container.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] cckellogg edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
cckellogg edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985073999


   > The tests now seem to be failing on `TestNacksTracker()`. Is that a consequence of this change, or an unrelated issue?
   
   It's probably unrelated since the tests passed for the other go versions. The concerning part is there may be a flaky `TestNacksTracker()` test now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   // since the client ticks at an interval of delay / 3
   // wait another interval to ensure we get all messages
   time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and `(*negativeAcksTracker).track()` also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions. Also the documentation says that `time.Ticker` is allowed to drop ticks, which I guess could be happening.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] sharashka-io commented on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
sharashka-io commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985372666


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and (*negativeAcksTracker).track() also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield commented on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   The concerning part is there may be a flaky TestNacksTracker() test now?
   
   I would guess that it's probably this part of newNackMockedConsumer():
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and (*negativeAcksTracker).track() also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield removed a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield removed a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-984693369


   I see this is now causing a max-message-size test to fail with an `AddToBatchFailed` error. Looking at the code, I think that's actually a legit failure mode when the message payload is too large, but would be great if someone like @wolfstudy had the time to take a look at some point.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield commented on pull request #678: [Issue 513] Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-994703309


   Made the additional tweak as suggested by @zzzming, also cleaned up `IsFull()` (which is not actually used anywhere as far as I can see).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield commented on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-984693369


   I see this is now causing a max-message-size test to fail with an `AddToBatchFailed` error. Looking at the code, I think that's actually a legit failure mode when the message payload is too large, but would be great if someone like @wolfstudy had the time to take a look at some point.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] cckellogg edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
cckellogg edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985073999


   > The tests now seem to be failing on `TestNacksTracker()`. Is that a consequence of this change, or an unrelated issue?
   
   It's probably unrelated since the tests passed for the other go versions. The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I started a retest


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985373428


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and `(*negativeAcksTracker).track()` also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] sharashka-io removed a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
sharashka-io removed a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985372666


   > The concerning part is there may be a flaky `TestNacksTracker()` test now?
   
   I would guess that it's probably this part of `newNackMockedConsumer()`:
   
   ```
   		// since the client ticks at an interval of delay / 3
   		// wait another interval to ensure we get all messages
   		time.Sleep(testNackDelay + 101*time.Millisecond)
   ```
   
   Since the sleep above happens inside a new goroutine, and (*negativeAcksTracker).track() also runs inside a new goroutine, it's probably possible in rare cases for the goroutine scheduling to interfere with the intended sequence of actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] bschofield edited a comment on pull request #678: Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
bschofield edited a comment on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-985394760


   I added `TestMaxBatchSize()` to `producer_test.go`, based on `TestMaxMessageSize()`.
   
   From the results of that test, it looks like the `<` in the size test should actually be `<=`, to accommodate the case where the new payload fits exactly in the container.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] zzzming commented on pull request #678: [Issue 513] Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
zzzming commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-991337168


   We should use this 
   ```
   bc.numMessages+1 <= bc.maxMessages 
   ```
   
   Because this will accommodate an individual message when batch is disabled. Currently this configuration fails at [unable to add message to batch]
   ```
   pulsar.ProducerOptions{
   		Topic:               topicName,
   		DisableBatching:     true,
   		BatchingMaxMessages: 1,
   		BatchingMaxSize:     1,
   	})
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar-client-go] zzzming commented on pull request #678: [Issue 513] Correct apparent logic error in batchContainer's hasSpace() method

Posted by GitBox <gi...@apache.org>.
zzzming commented on pull request #678:
URL: https://github.com/apache/pulsar-client-go/pull/678#issuecomment-991337168


   We should use this 
   ```
   bc.numMessages+1 <= bc.maxMessages 
   ```
   
   Because this will accommodate an individual message when batch is disabled. Currently this configuration fails at [unable to add message to batch]
   ```
   pulsar.ProducerOptions{
   		Topic:               topicName,
   		DisableBatching:     true,
   		BatchingMaxMessages: 1,
   		BatchingMaxSize:     1,
   	})
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org