You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "eye-gu (via GitHub)" <gi...@apache.org> on 2026/03/10 14:34:32 UTC

[PR] Support float64 in the streaming process [skywalking-banyandb]

eye-gu opened a new pull request, #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001

   <!--
       ⚠️ Please make sure to read this template first, pull requests that don't accord with this template
       maybe closed without notice.
       Texts surrounded by `<` and `>` are meant to be replaced by you, e.g. <framework name>, <issue number>.
       Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]`
   -->
   
   <!-- ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👇 ====
   ### <Feature description>
   - [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
   - [ ] Update the documentation to include this new feature.
   - [ ] Tests(including UT, IT, E2E) are added to verify the new feature.
   - [ ] If it's UI related, attach the screenshots below.
        ==== 🆕 Remove this line WHEN AND ONLY WHEN you're adding a new feature, follow the checklist 👆 ==== -->
   
   - [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Fixes apache/skywalking#<issue number>.
   - [ ] Update the [`CHANGES` log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#issuecomment-4035681612

   ## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `44.82759%` with `32 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 30.07%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`748026b`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/748026b4bc439839a61954c8adb08df99d61fd47?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 150 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [pkg/convert/number.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&filepath=pkg%2Fconvert%2Fnumber.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL2NvbnZlcnQvbnVtYmVyLmdv) | 0.00% | [16 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pkg/flow/streaming/topn.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&filepath=pkg%2Fflow%2Fstreaming%2Ftopn.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL2Zsb3cvc3RyZWFtaW5nL3RvcG4uZ28=) | 76.00% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pkg/query/logical/measure/measure\_top.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&filepath=pkg%2Fquery%2Flogical%2Fmeasure%2Fmeasure_top.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3F1ZXJ5L2xvZ2ljYWwvbWVhc3VyZS9tZWFzdXJlX3RvcC5nbw==) | 63.63% | [2 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../query/logical/measure/measure\_plan\_aggregation.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&filepath=pkg%2Fquery%2Flogical%2Fmeasure%2Fmeasure_plan_aggregation.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3F1ZXJ5L2xvZ2ljYWwvbWVhc3VyZS9tZWFzdXJlX3BsYW5fYWdncmVnYXRpb24uZ28=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pkg/query/logical/measure/measure\_plan\_top.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&filepath=pkg%2Fquery%2Flogical%2Fmeasure%2Fmeasure_plan_top.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3F1ZXJ5L2xvZ2ljYWwvbWVhc3VyZS9tZWFzdXJlX3BsYW5fdG9wLmdv) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [pkg/query/logical/measure/topn\_plan\_localscan.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&filepath=pkg%2Fquery%2Flogical%2Fmeasure%2Ftopn_plan_localscan.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL3F1ZXJ5L2xvZ2ljYWwvbWVhc3VyZS90b3BuX3BsYW5fbG9jYWxzY2FuLmdv) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (3530dd9) and HEAD (748026b). Click for more details.
   > 
   > <details><summary>HEAD has 1 upload less than BASE</summary>
   >
   >| Flag | BASE (3530dd9) | HEAD (748026b) |
   >|------|------|------|
   >||1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff             @@
   ##             main    #1001       +/-   ##
   ===========================================
   - Coverage   45.97%   30.07%   -15.91%     
   ===========================================
     Files         328      123      -205     
     Lines       55505    12877    -42628     
   ===========================================
   - Hits        25520     3873    -21647     
   + Misses      27909     8630    -19279     
   + Partials     2076      374     -1702     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.07% <44.82%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1001?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "eye-gu (via GitHub)" <gi...@apache.org>.
eye-gu commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r3036910617


##########
banyand/dquery/topn.go:
##########
@@ -103,6 +131,10 @@ func (t *topNQueryProcessor) Rev(ctx context.Context, message bus.Message) (resp
 			span.Stop()
 		}()
 	}
+	originalTopN := request.GetTopN()
+	// Set topN to 0 to disable truncation on data nodes for all aggregation functions.
+	// This ensures coordinator-side aggregation can see all relevant candidates.
+	request.TopN = 0

Review Comment:
   min/max can be truncated via TopN at the datanode, but sum/count/mean must read the full dataset to calculate accurate results. To avoid OOM, one approach is to follow a MapReduce pattern, but even then, sum/count/mean still need to collect all the pre-calculated data. If you agree, I can open a separate issue to fix this bug first, and then come back to this PR.



##########
banyand/dquery/topn.go:
##########
@@ -103,6 +131,10 @@ func (t *topNQueryProcessor) Rev(ctx context.Context, message bus.Message) (resp
 			span.Stop()
 		}()
 	}
+	originalTopN := request.GetTopN()
+	// Set topN to 0 to disable truncation on data nodes for all aggregation functions.
+	// This ensures coordinator-side aggregation can see all relevant candidates.
+	request.TopN = 0

Review Comment:
   > For high-cardinality entities, this could OOM data nodes during distributed TopN queries.
   
   min/max can be truncated via TopN at the datanode, but sum/count/mean must read the full dataset to calculate accurate results. To avoid OOM, one approach is to follow a MapReduce pattern, but even then, sum/count/mean still need to collect all the pre-calculated data. If you agree, I can open a separate issue to fix this bug first, and then come back to this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#issuecomment-4230409059

   > > @eye-gu I will close this. Feel free to reopen it if you would like to continue.
   > 
   > Okay.
   > 
   > The core issue right now is that setting TopN to 0 to fetch all data can lead to OOM. As I mentioned earlier, min/max can be safely truncated, but truncating count/sum/mean will produce incorrect results.
   > 
   > My thought is to implement TopN pushdown to the DataNode to reduce the data volume returned. I agree that this optimization shouldn't be included in this PR.
   > 
   > If you agree, I will open a separate issue to address this OOM problem first, and then come back to this PR later.
   
   
   The issue you mentioned is unrelated to this PR. Why do you switch to another issue?
   
   I have left a comment on the new issue. What you're concerned about does not seem to be a problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily closed pull request #1001: Support float64 in the streaming process
URL: https://github.com/apache/skywalking-banyandb/pull/1001


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "eye-gu (via GitHub)" <gi...@apache.org>.
eye-gu commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r2938487861


##########
banyand/measure/topn.go:
##########


Review Comment:
   Thank you for the valuable feedback. I have refactored the code to use Go's generic types for TopNValue and related components. And for float64 encoding, used encoding.Float64ListToDecimalIntList to convert float64 values to int64 before marshaling.
   Please let me know if you have any further suggestions or concerns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "eye-gu (via GitHub)" <gi...@apache.org>.
eye-gu commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r3036910617


##########
banyand/dquery/topn.go:
##########
@@ -103,6 +131,10 @@ func (t *topNQueryProcessor) Rev(ctx context.Context, message bus.Message) (resp
 			span.Stop()
 		}()
 	}
+	originalTopN := request.GetTopN()
+	// Set topN to 0 to disable truncation on data nodes for all aggregation functions.
+	// This ensures coordinator-side aggregation can see all relevant candidates.
+	request.TopN = 0

Review Comment:
   min/max can be truncated via TopN at the datanode, but sum/count/mean must read the full dataset to calculate accurate results. To avoid OOM, one approach is to follow a MapReduce pattern, but even then, sum/count/mean still need to collect all the pre-calculated data. If you agree, I can open a separate issue to fix this bug first, and then come back to this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r2978606124


##########
banyand/dquery/topn.go:
##########
@@ -103,6 +131,10 @@ func (t *topNQueryProcessor) Rev(ctx context.Context, message bus.Message) (resp
 			span.Stop()
 		}()
 	}
+	originalTopN := request.GetTopN()
+	// Set topN to 0 to disable truncation on data nodes for all aggregation functions.
+	// This ensures coordinator-side aggregation can see all relevant candidates.
+	request.TopN = 0

Review Comment:
   For high-cardinality entities, this could OOM data nodes during distributed TopN queries.



##########
pkg/flow/streaming/topn.go:
##########


Review Comment:
   Introduce a generic type:
   
   ```
   // TopSortKey is the constraint for TopN sort keys (int64 or float64).
   type TopSortKey interface {
   	int64 | float64
   }
   ```
   
   topNAggregatorGroup, TopNOption and etc should this type. 



##########
banyand/dquery/topn.go:
##########
@@ -46,6 +49,31 @@ type topNQueryProcessor struct {
 	*bus.UnImplementedHealthyListener
 }
 
+func (t *topNQueryProcessor) getTopNFieldType(ctx context.Context, group, topNName string) databasev1.FieldType {

Review Comment:
   It falls back silently to FIELD_TYPE_UNSPECIFIED → treated as int64. It's not correct.



##########
pkg/query/logical/measure/topn_plan_localscan.go:
##########
@@ -207,49 +207,78 @@ func (ei *topNMIterator) Next() bool {
 			return false
 		}
 		ts := timestamppb.New(time.Unix(0, r.Timestamps[i]))
-		topNValue.Reset()
-		err := topNValue.Unmarshal(bd, decoder)
-		if err != nil {
-			ei.err = multierr.Append(ei.err, errors.WithMessagef(err, "failed to unmarshal topN values[%d]:[%s]%s", i, ts, hex.EncodeToString(fv.GetBinaryData())))
+
+		fieldType, detectErr := measure.DetectFieldTypeFromBinary(bd)
+		if detectErr != nil {
+			ei.err = multierr.Append(ei.err, errors.WithMessagef(detectErr,
+				"failed to detect field type for topN values[%d]:[%s]%s", i, ts, hex.EncodeToString(fv.GetBinaryData())))
 			continue
 		}
-		shardID := uint32(0)
-		if i < len(r.ShardIDs) {
-			shardID = uint32(r.ShardIDs[i])
-		}
-		fieldName, entityNames, values, entities := topNValue.Values()
-		for j := range entities {
-			dp := &measurev1.DataPoint{
-				Timestamp: ts,
-				Sid:       uint64(r.SID),
-				Version:   r.Versions[i],
+
+		if fieldType == databasev1.FieldType_FIELD_TYPE_FLOAT {
+			topNValue := measure.GenerateTopNValueFloat()
+			defer measure.ReleaseTopNValueFloat(topNValue)
+			topNValue.Reset()
+			unmarshalErr := topNValue.Unmarshal(bd, decoder)
+			if unmarshalErr != nil {
+				ei.err = multierr.Append(ei.err, errors.WithMessagef(unmarshalErr, "failed to unmarshal topN values[%d]:[%s]%s", i, ts, hex.EncodeToString(fv.GetBinaryData())))
+				continue
 			}
-			tagFamily := &modelv1.TagFamily{
-				Name: measure.TopNTagFamily,
+			if procErr := processTopNValue(ei, topNValue, r, i, ts); procErr != nil {
+				ei.err = multierr.Append(ei.err, procErr)
 			}
-			dp.TagFamilies = append(dp.TagFamilies, tagFamily)
-			for k, entityName := range entityNames {
-				tagFamily.Tags = append(tagFamily.Tags, &modelv1.Tag{
-					Key:   entityName,
-					Value: entities[j][k],
-				})
+		} else {
+			topNValue := measure.GenerateTopNValueInt()
+			defer measure.ReleaseTopNValueInt(topNValue)

Review Comment:
   Move them outside the loop. Reuse the topNValue in the loop. defer's scope is the function not the for loop body. 



##########
banyand/dquery/topn.go:
##########
@@ -207,6 +215,53 @@ func (s *sortedTopNList) Val() *comparableTopNItem {
 	return &comparableTopNItem{s.Items[s.index-1]}
 }
 
+func processTopNResponse[N aggregation.Number](

Review Comment:
   In the old code, partial results from successful nodes were still used even when some nodes errored. Now, if 1 out of 10 nodes fails, the entire query returns an error. This is a regression for distributed scenarios.



##########
banyand/measure/topn.go:
##########
@@ -881,8 +1021,157 @@ func (t *TopNValue) marshal(dst []byte) ([]byte, error) {
 	return dst, nil
 }
 
+func (t *TopNValue[N]) marshalFloat64(dst []byte) ([]byte, error) {
+	dst = encoding.EncodeBytes(dst, convert.StringToBytes(t.valueName))
+	dst = encoding.VarUint64ToBytes(dst, uint64(len(t.entityTagNames)))
+	for _, entityTagName := range t.entityTagNames {
+		dst = encoding.EncodeBytes(dst, convert.StringToBytes(entityTagName))
+	}
+
+	valuesCount := uint64(len(t.values))
+	if valuesCount&(1<<63) != 0 {
+		return nil, errors.New("float values count overflow")
+	}
+	valuesCount |= (1 << 63)
+	dst = encoding.VarUint64ToBytes(dst, valuesCount)
+
+	floatValues := make([]float64, len(t.values))
+	for i, v := range t.values {
+		floatValues[i] = float64(v)
+	}
+
+	intValues, exponent, err := encoding.Float64ListToDecimalIntList(nil, floatValues)
+	if err != nil {
+		return nil, fmt.Errorf("failed to convert float64 to decimal int: %w", err)
+	}
+	t.exponent = exponent
+
+	t.buf, t.encodeType, t.firstValue = encoding.Int64ListToBytes(t.buf, intValues)
+	dst = append(dst, byte(t.encodeType))
+	dst = encoding.VarInt64ToBytes(dst, t.firstValue)
+	dst = encoding.VarInt64ToBytes(dst, int64(t.exponent))
+	dst = encoding.VarUint64ToBytes(dst, uint64(len(t.buf)))
+	dst = append(dst, t.buf...)
+
+	evv := t.resizeEntityValues(len(t.entities))
+	for i, tvv := range t.entities {
+		ev := evv[i]
+		ev, err = pbv1.MarshalTagValues(ev[:0], tvv)
+		if err != nil {
+			return nil, err
+		}
+		evv[i] = ev
+	}
+	dst = encoding.EncodeBytesBlock(dst, evv)
+	return dst, nil
+}
+
+// DetectFieldTypeFromBinary detects the field type from binary data.
+func DetectFieldTypeFromBinary(src []byte) (databasev1.FieldType, error) {
+	var err error
+	src, _, err = encoding.DecodeBytes(src)
+	if err != nil {
+		return databasev1.FieldType_FIELD_TYPE_UNSPECIFIED, err
+	}
+	var count uint64
+	src, count = encoding.BytesToVarUint64(src)
+	for i := uint64(0); i < count; i++ {
+		src, _, err = encoding.DecodeBytes(src)
+		if err != nil {
+			return databasev1.FieldType_FIELD_TYPE_UNSPECIFIED, err
+		}
+	}
+	_, count = encoding.BytesToVarUint64(src)
+	if (count & (1 << 63)) != 0 {
+		return databasev1.FieldType_FIELD_TYPE_FLOAT, nil
+	}
+	return databasev1.FieldType_FIELD_TYPE_INT, nil
+}
+
+// MergeTopNBinaryValues merges two TopN binary values and returns the merged binary value.
+func MergeTopNBinaryValues(
+	left, right []byte, topN int32, sort modelv1.Sort, decoder *encoding.BytesBlockDecoder,
+	timestamp uint64, leftVersion, rightVersion int64,
+) ([]byte, error) {
+	fieldType, detectErr := DetectFieldTypeFromBinary(left)
+	if detectErr != nil {

Review Comment:
   If left fails detection but right succeeds and returns FLOAT, then left (which might be int64 data from an older format) will be deserialized as float64 — causing data corruption.



##########
banyand/measure/topn_post_processor.go:
##########
@@ -207,24 +253,34 @@ func (taggr *topNPostProcessor) Put(entityValues pbv1.EntityValues, val int64, t
 		return
 	}
 
-	newItem := &topNAggregatorItem{
+	newItem := &topNAggregatorItem[N]{
 		val:     val,
 		key:     key,
 		values:  entityValues,
 		version: version,
 	}
 
+	if taggr.topN <= 0 {

Review Comment:
   This will cause OOM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r2918724400


##########
banyand/measure/topn.go:
##########


Review Comment:
   1. Utilize Go's generic types to implement topNStreamingProcessor, TopNValue, and related components. It is preferable to minimize the "switch" logic in this context.
   2. Convert []float to []int using encoding.Float64ListToDecimalIntList. Subsequently, marshal the int64 list following the existing logic. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "eye-gu (via GitHub)" <gi...@apache.org>.
eye-gu commented on PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#issuecomment-4334618077

   @hanahmily I have pushed new commits. Could you please reopen this PR and take another look? Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#issuecomment-4155182693

   @eye-gu Any update on this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "paultanay (via GitHub)" <gi...@apache.org>.
paultanay commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r3023169296


##########
banyand/dquery/topn.go:
##########
@@ -111,53 +143,26 @@ func (t *topNQueryProcessor) Rev(ctx context.Context, message bus.Message) (resp
 		return
 	}
 	var allErr error
-	aggregator := measure.CreateTopNPostProcessor(request.GetTopN(),
-		agg, request.GetFieldValueSort())
-	var tags []string
-	var responseCount int
-	for _, f := range ff {
-		if m, getErr := f.Get(); getErr != nil {
-			allErr = multierr.Append(allErr, getErr)
-		} else {
-			d := m.Data()
-			if d == nil {
-				continue
-			}
-			responseCount++
-			topNResp := d.(*measurev1.TopNResponse)
-			for _, l := range topNResp.Lists {
-				for _, tn := range l.Items {
-					if tags == nil {
-						tags = make([]string, 0, len(tn.Entity))
-						for _, e := range tn.Entity {
-							tags = append(tags, e.Key)
-						}
-					}
-					entityValues := make(pbv1.EntityValues, 0, len(tn.Entity))
-					for _, e := range tn.Entity {
-						entityValues = append(entityValues, e.Value)
-					}
-					aggregator.Put(entityValues, tn.Value.GetInt().GetValue(), uint64(tn.Timestamp.AsTime().UnixMilli()), tn.Version)
-				}
-			}
-		}
+	fieldType := t.getTopNFieldType(ctx, request.Groups[0], request.GetName())

Review Comment:
   I think there is still a robustness regression in `banyand/dquery/topn.go`: the new field-type lookup indexes `request.Groups[0]`, but `validateRequest()` still doesn’t check that `Groups` is non-empty.
   
   `TopNRequest` does declare `groups` with `min_items = 1`, so normal gRPC requests should usually be rejected earlier, but this path can still be reached outside the interceptor layer. Since the old distributed TopN path didn’t index `Groups[0]`, it looks safer to guard this explicitly here as well.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "eye-gu (via GitHub)" <gi...@apache.org>.
eye-gu commented on PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#issuecomment-4219092657

   > @eye-gu I will close this. Feel free to reopen it if you would like to continue.
   
   Okay.
   
   The core issue right now is that setting TopN to 0 to fetch all data can lead to OOM. As I mentioned earlier, min/max can be safely truncated, but truncating count/sum/mean will produce incorrect results. 
   
   My thought is to implement TopN pushdown to the DataNode to reduce the data volume returned. I agree that this optimization shouldn't be included in this PR. 
   
   If you agree, I will open a separate issue to address this OOM problem first, and then come back to this PR later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#issuecomment-4213870239

   @eye-gu I will close this. Feel free to reopen it if you would like to continue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r2915242698


##########
pkg/flow/streaming/topn.go:
##########
@@ -194,11 +200,30 @@ func (t *topNAggregatorGroup) getOrCreateGroup(group string) *topNAggregator {
 	t.aggregatorGroup[group] = &topNAggregator{
 		topNAggregatorGroup: t,
 		treeMap:             treemap.NewWith(t.comparator),
-		dict:                make(map[uint64]int64),
+		dict:                make(map[uint64]interface{}),
 	}
 	return t.aggregatorGroup[group]
 }
 
+func (t *topNAggregatorGroup) setComparatorFromFieldType() {
+	var baseComparator utils.Comparator
+	switch t.fieldType {
+	case databasev1.FieldType_FIELD_TYPE_INT:
+		baseComparator = utils.Int64Comparator
+	case databasev1.FieldType_FIELD_TYPE_FLOAT:
+		baseComparator = utils.Float64Comparator
+	default:
+		panic("unsupported field type: must be ValueTypeInt64 or ValueTypeFloat64")

Review Comment:
   `setComparatorFromFieldType` panics on unsupported `FieldType`. Since `TopN()` is part of the streaming pipeline setup, a panic here can crash the process from a misconfiguration or schema evolution. Consider converting this to an error path (e.g., `drainErr`) and make the message reflect actual `databasev1.FieldType` values (the current text mentions `ValueTypeInt64/Float64`).
   ```suggestion
   		// Unsupported field type: log and fall back to a no-op comparator instead of panicking.
   		if t.l != nil {
   			t.l.Error().
   				Interface("fieldType", t.fieldType).
   				Msg("unsupported field type for TopN; defaulting to no-op comparator")
   		}
   		baseComparator = func(a, b interface{}) int {
   			// Treat all values as equal to avoid type assumptions and panics.
   			return 0
   		}
   ```



##########
banyand/dquery/topn.go:
##########
@@ -180,7 +182,17 @@ type comparableTopNItem struct {
 	*measurev1.TopNList_Item
 }
 
+func truncationDataNode(request *measurev1.TopNRequest) {
+	// Set topN to 0 to disable truncation on data nodes
+	if request.Agg != modelv1.AggregationFunction_AGGREGATION_FUNCTION_MAX {
+		request.TopN = 0
+	}
+}
+

Review Comment:
   `truncationDataNode` keeps per-node truncation for `AGGREGATION_FUNCTION_MAX`, but coordinator-side aggregation still depends on seeing candidates that may be outside any single node’s local TopN. This can produce incorrect global TopN results (entities can be dropped on all nodes due to local competition). If the goal is correctness for distributed aggregation, consider disabling truncation for *all* aggregation functions (or over-fetching more than `TopN` per node).
   ```suggestion
   	// Set topN to 0 to disable truncation on data nodes for all aggregation functions.
   	// This ensures coordinator-side aggregation can see all relevant candidates.
   	request.TopN = 0
   }
   ```



##########
banyand/measure/topn.go:
##########
@@ -678,26 +717,27 @@ func (manager *topNProcessorManager) buildFilter(criteria *modelv1.Criteria) (fl
 	}, nil
 }
 
-func (manager *topNProcessorManager) buildMapper(fieldName string, groupByNames ...string) (flow.UnaryFunc[any], error) {
+func (manager *topNProcessorManager) buildMapper(fieldName string, groupByNames ...string) (flow.UnaryFunc[any], databasev1.FieldType, error) {
 	fieldIdx := slices.IndexFunc(manager.m.GetFields(), func(spec *databasev1.FieldSpec) bool {
 		return spec.GetName() == fieldName
 	})
 	if fieldIdx == -1 {
 		manager.l.Warn().Str("fieldName", fieldName).Str("measure", manager.m.Metadata.GetName()).
 			Msg("TopNAggregation references removed field which no longer exists in schema, ignoring this TopNAggregation")
-		return nil, fmt.Errorf("field %s is not found in %s schema", fieldName, manager.m.Metadata.GetName())
+		return nil, databasev1.FieldType_FIELD_TYPE_UNSPECIFIED, fmt.Errorf("field %s is not found in %s schema", fieldName, manager.m.Metadata.GetName())
 	}
+	fieldType := manager.m.GetFields()[fieldIdx].GetFieldType()
 	if len(groupByNames) == 0 {
 		return func(_ context.Context, request any) any {
 			dpWithEvs := request.(*dataPointWithEntityValues)
 			return flow.Data{
 				dpWithEvs.entityValues,
 				"",
-				dpWithEvs.intFieldValue(fieldName, manager.l),
+				dpWithEvs.fieldValue(fieldName, fieldType, manager.l),

Review Comment:
   `buildMapper` returns the field’s `FieldType` but doesn’t validate that it’s supported by the streaming TopN comparator (currently INT/FLOAT only). If the field is another type (or later extended), this will either silently map to `intFieldValue` (returning 0) or trigger a panic downstream when selecting the comparator. Consider explicitly validating `fieldType` here and returning an error for unsupported types.



##########
banyand/measure/topn.go:
##########
@@ -956,6 +1102,19 @@ func (t *TopNValue) Unmarshal(src []byte, decoder *encoding.BytesBlockDecoder) e
 	return nil
 }
 
+func (t *TopNValue) bytesToFloat64List(dst []float64, src []byte, count int) []float64 {
+	dst = dst[:0]
+	for i := 0; i < count; i++ {
+		if len(src) < 8 {
+			break
+		}
+		v := convert.BytesToFloat64(src[:8])
+		dst = append(dst, v)
+		src = src[8:]

Review Comment:
   `bytesToFloat64List` silently truncates if `src` is shorter than `count*8` (it `break`s), which can mask data corruption and lead to mismatched `valuesCount` vs decoded values/entities. Consider validating `valueLen == valuesCount*8` (and that enough bytes exist) and returning an error instead of returning a shorter slice.



##########
banyand/measure/topn_post_processor.go:
##########
@@ -160,26 +298,28 @@ type topNPostProcessor struct {
 	topN      int32
 }
 
-func (taggr *topNPostProcessor) Put(entityValues pbv1.EntityValues, val int64, timestampMillis uint64, version int64) {
+func (taggr *topNPostProcessor) Put(entityValues pbv1.EntityValues, val SortableValue, timestampMillis uint64, version int64) {
 	timeline, ok := taggr.timelines[timestampMillis]
 	key := entityValues.String()
 	if !ok {
 		timeline = &topNTimelineItem{

Review Comment:
   `Put` currently doesn't treat `topN <= 0` as "no truncation": with `topN==0` the first item is still pushed and subsequent inserts are effectively limited to 1 element (because `Len() < int(taggr.topN)` is always false). This breaks the distributed-query workaround that sets `TopN=0` to disable truncation. Consider adding an early branch for `taggr.topN <= 0` to always accept all items (no ReplaceLowest/eviction) or define a separate explicit sentinel for "unbounded".



##########
banyand/measure/topn.go:
##########
@@ -881,6 +932,50 @@ func (t *TopNValue) marshal(dst []byte) ([]byte, error) {
 	return dst, nil
 }
 
+func (t *TopNValue) marshalFloat64(dst []byte) ([]byte, error) {
+	dst = encoding.EncodeBytes(dst, convert.StringToBytes(t.valueName))
+	dst = encoding.VarUint64ToBytes(dst, uint64(len(t.entityTagNames)))
+	for _, entityTagName := range t.entityTagNames {
+		dst = encoding.EncodeBytes(dst, convert.StringToBytes(entityTagName))
+	}
+
+	valuesCount := uint64(len(t.values))
+	if valuesCount&(1<<63) != 0 {
+		return nil, errors.New("float values count overflow")
+	}
+	valuesCount |= (1 << 63)
+	dst = encoding.VarUint64ToBytes(dst, valuesCount)
+
+	floatValues := make([]float64, len(t.values))
+	for i, v := range t.values {
+		floatValues[i] = float64(v.(FloatValue))
+	}
+	t.buf = t.float64ListToBytes(t.buf, floatValues)
+	dst = encoding.VarUint64ToBytes(dst, uint64(len(t.buf)))
+	dst = append(dst, t.buf...)
+
+	var err error
+	evv := t.resizeEntityValues(len(t.entities))
+	for i, tvv := range t.entities {
+		ev := evv[i]
+		ev, err = pbv1.MarshalTagValues(ev[:0], tvv)
+		if err != nil {
+			return nil, err
+		}
+		evv[i] = ev
+	}
+	dst = encoding.EncodeBytesBlock(dst, evv)
+	return dst, nil
+}
+
+func (t *TopNValue) float64ListToBytes(dst []byte, values []float64) []byte {
+	dst = dst[:0]
+	for _, v := range values {
+		dst = append(dst, convert.Float64ToBytes(v)...)
+	}

Review Comment:
   `float64ListToBytes` currently appends `convert.Float64ToBytes(v)...` for each value, which allocates a new 8-byte slice per element and can be hot in TopN streaming. Consider pre-growing `dst` to `len(values)*8` and writing in-place with `binary.BigEndian.PutUint64` to avoid per-value allocations.



##########
pkg/flow/streaming/topn.go:
##########
@@ -61,13 +62,10 @@ func (s *windowedFlow) TopN(topNum int, opts ...any) flow.Flow {
 		if topNAggrFunc.sortKeyExtractor == nil {
 			s.f.drainErr(errors.New("sortKeyExtractor must be specified"))
 		}
-		if topNAggrFunc.sort == ASC {
-			topNAggrFunc.comparator = utils.Int64Comparator
-		} else { // DESC
-			topNAggrFunc.comparator = func(a, b interface{}) int {
-				return utils.Int64Comparator(b, a)
-			}
+		if topNAggrFunc.fieldType == databasev1.FieldType_FIELD_TYPE_UNSPECIFIED {
+			s.f.drainErr(errors.New("fieldType must be specified"))
 		}
+		topNAggrFunc.setComparatorFromFieldType()

Review Comment:
   In `windowedFlow.TopN`, after calling `drainErr` for missing `sortKeyExtractor` / `fieldType`, the code continues and unconditionally calls `setComparatorFromFieldType()`. If `fieldType` is still `FIELD_TYPE_UNSPECIFIED`, this will panic. Consider returning early after draining the error (or guarding `setComparatorFromFieldType` so it never panics on invalid config).
   ```suggestion
   		} else {
   			topNAggrFunc.setComparatorFromFieldType()
   		}
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Support float64 in the streaming process [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #1001:
URL: https://github.com/apache/skywalking-banyandb/pull/1001#discussion_r3031346510


##########
banyand/dquery/topn.go:
##########
@@ -111,53 +143,26 @@ func (t *topNQueryProcessor) Rev(ctx context.Context, message bus.Message) (resp
 		return
 	}
 	var allErr error
-	aggregator := measure.CreateTopNPostProcessor(request.GetTopN(),
-		agg, request.GetFieldValueSort())
-	var tags []string
-	var responseCount int
-	for _, f := range ff {
-		if m, getErr := f.Get(); getErr != nil {
-			allErr = multierr.Append(allErr, getErr)
-		} else {
-			d := m.Data()
-			if d == nil {
-				continue
-			}
-			responseCount++
-			topNResp := d.(*measurev1.TopNResponse)
-			for _, l := range topNResp.Lists {
-				for _, tn := range l.Items {
-					if tags == nil {
-						tags = make([]string, 0, len(tn.Entity))
-						for _, e := range tn.Entity {
-							tags = append(tags, e.Key)
-						}
-					}
-					entityValues := make(pbv1.EntityValues, 0, len(tn.Entity))
-					for _, e := range tn.Entity {
-						entityValues = append(entityValues, e.Value)
-					}
-					aggregator.Put(entityValues, tn.Value.GetInt().GetValue(), uint64(tn.Timestamp.AsTime().UnixMilli()), tn.Version)
-				}
-			}
-		}
+	fieldType := t.getTopNFieldType(ctx, request.Groups[0], request.GetName())

Review Comment:
   It makes sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org