You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "ButterBright (via GitHub)" <gi...@apache.org> on 2026/05/11 16:10:47 UTC

[PR] Support multiple types for the same tag in a single measure or stream… [skywalking-banyandb]

ButterBright opened a new pull request, #1126:
URL: https://github.com/apache/skywalking-banyandb/pull/1126

   … part
   
   <!--
       ⚠️ Please make sure to read this template first, pull requests that don't accord with this template
       maybe closed without notice.
       Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
       Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
   -->
   
   <!-- ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👇 ====
   ### Fix <bug description or the bug issue number or bug issue link>
   - [ ] Add a unit test to verify that the fix works.
   - [ ] Explain briefly why the bug exists and how to fix it.
        ==== 🐛 Remove this line WHEN AND ONLY WHEN you're fixing a bug, follow the checklist 👆 ==== -->
   
   <!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
   ### <Feature description>
   - [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
   - [ ] Update the documentation to include this new feature.
   - [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
   - [ ] If it's UI related, attach the screenshots below.
        ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support multiple types for the same tag in a single measure or stream part [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #1126:
URL: https://github.com/apache/skywalking-banyandb/pull/1126#discussion_r3223659090


##########
banyand/measure/block.go:
##########
@@ -956,6 +968,10 @@ func fastTagAppend(bi, b *blockPointer, offset int) error {
 				return fmt.Errorf("unexpected tag name for tag family %q: got %q; want %q",
 					bi.tagFamilies[i].name, b.tagFamilies[i].columns[j].name, bi.tagFamilies[i].columns[j].name)
 			}
+		}
+	}
+	for i := range bi.tagFamilies {

Review Comment:
   The original `fastTagAppend` performed the name check and the append inside the same loop. The change splits it into two full passes over the same tag families × columns with no functional reason — renaming happens on the source block before this function is called, so column names are stable here. The split is extra work and creates two places that must stay in sync. Mirror in `banyand/stream/block.go`.



##########
banyand/measure/merger.go:
##########
@@ -359,6 +366,101 @@ func mergeBlocks(closeCh <-chan struct{}, bw *blockWriter, br *blockReader) (*pa
 	return &result, nil
 }
 
+func renameConflictColumns(b *block, conflictColumns map[string]map[string]struct{}) {
+	if len(conflictColumns) == 0 {
+		return
+	}
+	for i := range b.tagFamilies {
+		columns := conflictColumns[b.tagFamilies[i].name]
+		if columns == nil {
+			continue
+		}
+		cc := b.tagFamilies[i].columns
+		for j := range cc {
+			if _, ok := columns[cc[j].name]; ok {
+				cc[j].name = encodeTypedColumn(cc[j].name, cc[j].valueType)
+			}
+		}
+	}
+}
+
+func collectConflictColumns(parts []*partWrapper) map[string]map[string]struct{} {
+	familyColumnTypes := make(map[string]map[string]map[pbv1.ValueType]struct{})
+	for _, pw := range parts {
+		partTypes := readPartColumnTypes(pw.p)
+		for cf, cc := range partTypes {
+			columnTypes := familyColumnTypes[cf]
+			if columnTypes == nil {
+				columnTypes = make(map[string]map[pbv1.ValueType]struct{})
+				familyColumnTypes[cf] = columnTypes
+			}
+			for name, vt := range cc {
+				decoded := decodeTypedColumn(name)
+				valueTypes := columnTypes[decoded]
+				if valueTypes == nil {
+					valueTypes = make(map[pbv1.ValueType]struct{})
+					columnTypes[decoded] = valueTypes
+				}
+				valueTypes[vt] = struct{}{}
+			}
+		}
+	}
+	var conflictColumns map[string]map[string]struct{}
+	for cf, columnTypes := range familyColumnTypes {
+		for name, valueTypes := range columnTypes {
+			if len(valueTypes) <= 1 {
+				continue
+			}
+			if conflictColumns == nil {
+				conflictColumns = make(map[string]map[string]struct{})
+			}
+			if conflictColumns[cf] == nil {
+				conflictColumns[cf] = make(map[string]struct{})
+			}
+			conflictColumns[cf][name] = struct{}{}
+		}
+	}
+	return conflictColumns
+}
+
+func readPartColumnTypes(p *part) map[string]map[string]pbv1.ValueType {
+	if len(p.tagFamilyMetadata) == 0 || len(p.primaryBlockMetadata) == 0 {
+		return nil
+	}
+	pmi := generatePartMergeIter()
+	defer releasePartMergeIter(pmi)
+	pmi.mustInitFromPart(p)
+
+	result := make(map[string]map[string]pbv1.ValueType)
+	bb := bigValuePool.Generate()
+	defer bigValuePool.Release(bb)
+	for pmi.nextBlockMetadata() {
+		for cf, db := range pmi.block.bm.tagFamilies {
+			reader, ok := p.tagFamilyMetadata[cf]
+			if !ok {
+				continue
+			}
+			bb.Buf = bytes.ResizeExact(bb.Buf, int(db.size))
+			fs.MustReadData(reader, int64(db.offset), bb.Buf)
+			cfm := generateColumnFamilyMetadata()
+			if _, err := cfm.unmarshal(bb.Buf); err != nil {

Review Comment:
   Unmarshal failure is silently swallowed. If a part's column-family metadata block is corrupted, conflict detection skips the column, the merger doesn't rename it, and the resulting merged part ends up with the exact duplicate-typed columns this change is meant to prevent — without any signal that something went wrong. At minimum this should log a warning. Same in `banyand/stream/merger.go`.



##########
banyand/measure/column.go:
##########
@@ -27,6 +29,57 @@ import (
 	"github.com/apache/skywalking-banyandb/pkg/pool"
 )
 
+const typedColumnSeparator = "#"

Review Comment:
   This introduces `#` as a reserved separator in on-disk column names. Please confirm that the schema validator forbids `#` in user-provided tag names. If a tag named `foo#int` is allowed, `decodeTypedColumn` will silently strip the suffix to `foo`, which then causes `collectConflictColumns` to manufacture a phantom conflict and the merger to rename a non-conflicting column. Same applies to `banyand/stream/tag.go`.



##########
banyand/measure/column.go:
##########
@@ -27,6 +29,57 @@ import (
 	"github.com/apache/skywalking-banyandb/pkg/pool"
 )
 
+const typedColumnSeparator = "#"
+
+var (
+	valueTypeToSuffix = map[pbv1.ValueType]string{
+		pbv1.ValueTypeStr:        "str",
+		pbv1.ValueTypeInt64:      "int",
+		pbv1.ValueTypeFloat64:    "float",
+		pbv1.ValueTypeBinaryData: "bin",
+		pbv1.ValueTypeStrArr:     "str_arr",
+		pbv1.ValueTypeInt64Arr:   "int_arr",
+		pbv1.ValueTypeTimestamp:  "ts",
+		pbv1.ValueTypeUnknown:    "",
+	}
+	suffixToValueType = map[string]pbv1.ValueType{
+		"str":     pbv1.ValueTypeStr,
+		"int":     pbv1.ValueTypeInt64,
+		"float":   pbv1.ValueTypeFloat64,
+		"bin":     pbv1.ValueTypeBinaryData,
+		"str_arr": pbv1.ValueTypeStrArr,
+		"int_arr": pbv1.ValueTypeInt64Arr,
+		"ts":      pbv1.ValueTypeTimestamp,
+	}
+)
+
+func encodeTypedColumn(name string, vt pbv1.ValueType) string {
+	suffix, ok := valueTypeToSuffix[vt]
+	if !ok || suffix == "" {
+		return name
+	}
+	return name + typedColumnSeparator + suffix
+}
+
+func decodeTypedColumn(key string) string {
+	for suffix := range suffixToValueType {
+		sepSuffix := typedColumnSeparator + suffix
+		if strings.HasSuffix(key, sepSuffix) {
+			return key[:len(key)-len(sepSuffix)]
+		}
+	}
+	return key
+}
+
+func hasTypeSuffix(key string) bool {

Review Comment:
   `hasTypeSuffix` (and its unit test) is added here but is never called from `banyand/measure` — same for the stream counterpart in `banyand/stream/tag.go`. The only callers live in `banyand/trace` and `banyand/internal/sidx`. Either remove the dead copies from measure/stream or wire them into the reader the way trace does.



##########
banyand/measure/block.go:
##########
@@ -225,13 +226,24 @@ func (b *block) unmarshalTagFamily(decoder *encoding.BytesBlockDecoder, tfIndex
 	cc := b.tagFamilies[tfIndex].resizeColumns(len(tagProjection))
 NEXT:
 	for j := range tagProjection {
+		tp := tagProjection[j]
+		if schemaType, ok := schemaTagTypes[tp]; ok {
+			typedName := encodeTypedColumn(tp, schemaType)
+			for i := range cfm.columnMetadata {
+				if cfm.columnMetadata[i].name == typedName {
+					cc[j].mustReadValues(decoder, valueReader, cfm.columnMetadata[i], uint64(b.Len()))
+					cc[j].name = tp
+					continue NEXT
+				}
+			}
+		}
 		for i := range cfm.columnMetadata {

Review Comment:
   After the typed-name lookup misses, the fallback matches on the plain name without checking that the on-disk `valueType` agrees with the schema type. The four downstream output sites in this file currently filter out wrong-typed columns with `c.valueType == schemaType` guards, so the user-visible result is correct today — but the contract here is fragile: every future read site that consumes `cf.columns[i].values` must remember the rule. Compare `banyand/trace/block.go` where the projection planner gates the match on `bm.tagType[typedTag] == schemaType` once at read time, so wrong-typed columns simply don't appear in the cursor. Mirror in `banyand/stream/block.go`.



##########
banyand/measure/merger.go:
##########
@@ -359,6 +366,101 @@ func mergeBlocks(closeCh <-chan struct{}, bw *blockWriter, br *blockReader) (*pa
 	return &result, nil
 }
 
+func renameConflictColumns(b *block, conflictColumns map[string]map[string]struct{}) {
+	if len(conflictColumns) == 0 {
+		return
+	}
+	for i := range b.tagFamilies {
+		columns := conflictColumns[b.tagFamilies[i].name]
+		if columns == nil {
+			continue
+		}
+		cc := b.tagFamilies[i].columns
+		for j := range cc {
+			if _, ok := columns[cc[j].name]; ok {
+				cc[j].name = encodeTypedColumn(cc[j].name, cc[j].valueType)
+			}
+		}
+	}
+}
+
+func collectConflictColumns(parts []*partWrapper) map[string]map[string]struct{} {
+	familyColumnTypes := make(map[string]map[string]map[pbv1.ValueType]struct{})
+	for _, pw := range parts {
+		partTypes := readPartColumnTypes(pw.p)
+		for cf, cc := range partTypes {
+			columnTypes := familyColumnTypes[cf]
+			if columnTypes == nil {
+				columnTypes = make(map[string]map[pbv1.ValueType]struct{})
+				familyColumnTypes[cf] = columnTypes
+			}
+			for name, vt := range cc {
+				decoded := decodeTypedColumn(name)
+				valueTypes := columnTypes[decoded]
+				if valueTypes == nil {
+					valueTypes = make(map[pbv1.ValueType]struct{})
+					columnTypes[decoded] = valueTypes
+				}
+				valueTypes[vt] = struct{}{}
+			}
+		}
+	}
+	var conflictColumns map[string]map[string]struct{}
+	for cf, columnTypes := range familyColumnTypes {
+		for name, valueTypes := range columnTypes {
+			if len(valueTypes) <= 1 {
+				continue
+			}
+			if conflictColumns == nil {
+				conflictColumns = make(map[string]map[string]struct{})
+			}
+			if conflictColumns[cf] == nil {
+				conflictColumns[cf] = make(map[string]struct{})
+			}
+			conflictColumns[cf][name] = struct{}{}
+		}
+	}
+	return conflictColumns
+}
+
+func readPartColumnTypes(p *part) map[string]map[string]pbv1.ValueType {

Review Comment:
   This function is called from `collectConflictColumns` for every input part on every merge. It walks every block metadata in the part and, for each block, issues `fs.MustReadData` for each tag family and unmarshals the column-family metadata — purely to detect whether the merge has a type conflict. The very next pass (`mergeBlocks`) then re-reads the same metadata when blocks are loaded. In the common case where the schema hasn't changed, the result is an empty conflict set and the entire scan is wasted IO. Worth thinking about whether the per-part column→type summary can be persisted (the trace and sidx variants already do this — see `banyand/trace/merger.go` where `pw.p.tagType` is read directly) or at least cached on the `part` after open. Mirror in `banyand/stream/merger.go`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support multiple types for the same tag in a single measure or stream part [skywalking-banyandb]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1126:
URL: https://github.com/apache/skywalking-banyandb/pull/1126#issuecomment-4423694032

   ## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1126?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 32.69%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e8b598c`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/e8b598cb87179632ef8540a0281348c6b61238d5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 249 commits behind head on main.
   > :exclamation:  There is a different number of reports uploaded between BASE (3530dd9) and HEAD (e8b598c). Click for more details.
   > 
   > <details><summary>HEAD has 1 upload less than BASE</summary>
   >
   >| Flag | BASE (3530dd9) | HEAD (e8b598c) |
   >|------|------|------|
   >||1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##             main    #1126       +/-   ##
   ===========================================
   - Coverage   45.97%   32.69%   -13.29%     
   ===========================================
     Files         328      131      -197     
     Lines       55505    14023    -41482     
   ===========================================
   - Hits        25520     4585    -20935     
   + Misses      27909     9019    -18890     
   + Partials     2076      419     -1657     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1126/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1126/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.69% <ø> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1126?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support multiple types for the same tag in a single measure or stream part [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #1126:
URL: https://github.com/apache/skywalking-banyandb/pull/1126


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org