You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by GitBox <gi...@apache.org> on 2020/03/26 06:53:49 UTC
[GitHub] [samza] bkonold opened a new pull request #1331: SAMZA-2495:
InMemorySystemAdmin offsets off by one when END_OF_STREAM present
bkonold opened a new pull request #1331: SAMZA-2495: InMemorySystemAdmin offsets off by one when END_OF_STREAM present
URL: https://github.com/apache/samza/pull/1331
I found this when refactoring side input restore to use RunLoop (https://github.com/apache/samza/pull/1330); this patch is necessary to prevent integration tests from failing after that refactor as marking side inputs as caught relies on newest offset.
**Symptom:** When an end of stream message is present in an in memory SSP, the new and upcoming offset are reported as 1 higher than they should be.
Example:
| E1, offset: 0 | E2, offset: 1 | ... | E4, offset: 3 | EndOfStream |
In the above case, InMemoryManager will report a newestOffset of 4 and an upcomingOffset of 5; they should be 3 and 4 respectively.
**Cause:** InMemoryManager treats end of stream as any other envelope with a numeric offset even though it is not.
**Changes:** Check in `InMemoryManager.constructSystemStreamMetadata` whether the last message in the stream is an EndOfStream, and adjust offsets accordingly.
**Tests:** Added a unit test to verify that offsets are adjusted by 1 in the case of EndOfStream.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] bkonold commented on issue #1331: SAMZA-2495:
InMemorySystemAdmin offsets off by one when END_OF_STREAM present
Posted by GitBox <gi...@apache.org>.
bkonold commented on issue #1331: SAMZA-2495: InMemorySystemAdmin offsets off by one when END_OF_STREAM present
URL: https://github.com/apache/samza/pull/1331#issuecomment-604634892
This may not be the right fix. Discarding for now while I spend more time investigating how end of stream is used
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [samza] bkonold closed pull request #1331: SAMZA-2495:
InMemorySystemAdmin offsets off by one when END_OF_STREAM present
Posted by GitBox <gi...@apache.org>.
bkonold closed pull request #1331: SAMZA-2495: InMemorySystemAdmin offsets off by one when END_OF_STREAM present
URL: https://github.com/apache/samza/pull/1331
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services