You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "ButterBright (via GitHub)" <gi...@apache.org> on 2024/01/24 06:58:02 UTC

[PR] Remove maxBlockLength in stream [skywalking-banyandb]

ButterBright opened a new pull request, #381:
URL: https://github.com/apache/skywalking-banyandb/pull/381

   <!--
       ⚠️ Please make sure to read this template first, pull requests that don't accord with this template
       maybe closed without notice.
       Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
       Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
   -->
   
   <!-- ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👇 ====
   ### Fix <bug description or the bug issue number or bug issue link>
   - [ ] Add a unit test to verify that the fix works.
   - [ ] Explain briefly why the bug exists and how to fix it.
        ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👆 ==== -->
   
   <!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
   ### <Feature description>
   - [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
   - [ ] Update the documentation to include this new feature.
   - [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
   - [ ] If it's UI related, attach the screenshots below.
        ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove maxBlockLength in stream [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #381:
URL: https://github.com/apache/skywalking-banyandb/pull/381#discussion_r1464949063


##########
banyand/stream/merger.go:
##########
@@ -300,31 +300,15 @@ func mergeBlocks(closeCh <-chan struct{}, bw *blockWriter, br *blockReader) (*pa
 		tmpBlock.bm.seriesID = b.bm.seriesID
 		br.loadBlockData(getDecoder())
 		mergeTwoBlocks(tmpBlock, pendingBlock, b)
-		if len(tmpBlock.timestamps) <= maxBlockLength && tmpBlock.uncompressedSizeBytes() <= maxUncompressedBlockSize {
+		if tmpBlock.uncompressedSizeBytes() <= maxUncompressedBlockSize {
 			if len(tmpBlock.timestamps) == 0 {
 				pendingBlockIsEmpty = true
 			}
 			pendingBlock, tmpBlock = tmpBlock, pendingBlock
-			continue
-		}
-
-		if len(tmpBlock.timestamps) <= maxBlockLength {
+		} else {
 			bw.mustWriteBlock(tmpBlock.bm.seriesID, &tmpBlock.block)

Review Comment:
   The "pendingBlock" and "tmpBlock" should be reset after writing. "pendingBlockIsEmpty" should be set to "true"



##########
banyand/stream/merger.go:
##########
@@ -300,31 +300,15 @@ func mergeBlocks(closeCh <-chan struct{}, bw *blockWriter, br *blockReader) (*pa
 		tmpBlock.bm.seriesID = b.bm.seriesID
 		br.loadBlockData(getDecoder())
 		mergeTwoBlocks(tmpBlock, pendingBlock, b)
-		if len(tmpBlock.timestamps) <= maxBlockLength && tmpBlock.uncompressedSizeBytes() <= maxUncompressedBlockSize {
+		if tmpBlock.uncompressedSizeBytes() <= maxUncompressedBlockSize {
 			if len(tmpBlock.timestamps) == 0 {
 				pendingBlockIsEmpty = true
 			}
 			pendingBlock, tmpBlock = tmpBlock, pendingBlock
-			continue
-		}
-
-		if len(tmpBlock.timestamps) <= maxBlockLength {
+		} else {

Review Comment:
   remove "else", add "continue" at the end of "if"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove maxBlockLength in stream [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #381:
URL: https://github.com/apache/skywalking-banyandb/pull/381#discussion_r1465897270


##########
banyand/stream/part_metadata.go:
##########
@@ -32,8 +32,7 @@ type partMetadata struct {
 	BlocksCount           uint64 `json:"blocksCount"`
 	MinTimestamp          int64  `json:"minTimestamp"`
 	MaxTimestamp          int64  `json:"maxTimestamp"`
-	// TODO: remove this?
-	ID uint64 `json:"-"`
+	ID                    uint64 `json:"-"`

Review Comment:
   You should raise another PR to remove it and the relevant "partID" field since the stream doesn't have to duplicate data.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove maxBlockLength in stream [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #381:
URL: https://github.com/apache/skywalking-banyandb/pull/381#discussion_r1464399046


##########
banyand/stream/part.go:
##########
@@ -325,8 +325,7 @@ func (mp *memPart) mustInitFromElements(es *elements) {
 			sidPrev = sid
 		}
 
-		if uncompressedBlockSizeBytes >= maxUncompressedBlockSize ||
-			(i-indexPrev) > maxBlockLength || sid != sidPrev {
+		if uncompressedBlockSizeBytes >= maxUncompressedBlockSize || sid != sidPrev {

Review Comment:
   You should remove "indexPrev" as well



##########
banyand/stream/merger.go:
##########
@@ -300,20 +300,14 @@ func mergeBlocks(closeCh <-chan struct{}, bw *blockWriter, br *blockReader) (*pa
 		tmpBlock.bm.seriesID = b.bm.seriesID
 		br.loadBlockData(getDecoder())
 		mergeTwoBlocks(tmpBlock, pendingBlock, b)
-		if len(tmpBlock.timestamps) <= maxBlockLength && tmpBlock.uncompressedSizeBytes() <= maxUncompressedBlockSize {
+		if tmpBlock.uncompressedSizeBytes() <= maxUncompressedBlockSize {
 			if len(tmpBlock.timestamps) == 0 {
 				pendingBlockIsEmpty = true
 			}
 			pendingBlock, tmpBlock = tmpBlock, pendingBlock
 			continue
 		}
 
-		if len(tmpBlock.timestamps) <= maxBlockLength {
-			bw.mustWriteBlock(tmpBlock.bm.seriesID, &tmpBlock.block)
-			releaseDecoder()
-			continue
-		}
-		tmpBlock.idx = maxBlockLength

Review Comment:
   You can't simply remove it.
   
   When the tmpBlock's size is larger than maxUncompressedBlockSize, it should be written. The rest of the pieces in the loop will be unnecessary. 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove maxBlockLength in stream [skywalking-banyandb]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #381:
URL: https://github.com/apache/skywalking-banyandb/pull/381#issuecomment-1907489247

   ## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/381?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base [(`97e9111`)](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/97e911121c4f9ec7ad07ac1580f50b39a03de644?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 47.31% compared to head [(`98e64d8`)](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/381?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 47.34%.
   > Report is 1 commits behind head on main.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main     #381      +/-   ##
   ==========================================
   + Coverage   47.31%   47.34%   +0.02%     
   ==========================================
     Files         172      172              
     Lines       21296    21290       -6     
   ==========================================
   + Hits        10077    10079       +2     
   + Misses      10377    10370       -7     
   + Partials      842      841       -1     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/381?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Remove maxBlockLength in stream [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #381:
URL: https://github.com/apache/skywalking-banyandb/pull/381


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org