You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by lu...@apache.org on 2022/08/25 08:12:10 UTC
[skywalking-banyandb] branch simple-streaming-api updated (9d1ca8f -> 10dbd18)
This is an automated email from the ASF dual-hosted git repository.
lujiajing pushed a change to branch simple-streaming-api
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
omit 9d1ca8f fix ci
omit c4337ef fix test cases
omit dba2341 fix LICENSE
omit 9bb280f fix lint
omit 665cdfc change window impl
omit a8b5221 fix data race
omit 39ec7af refactor open api
omit 314d427 fix test
omit 4eec9bd refactor flow error handling
omit 49a3aae refactor timeTrigger and triggerContext
omit b7c14f4 change Enum typing and change watermark strategy
omit 0c9b6a6 refactor flow API
omit 57eb398 rename pq
omit 67346b9 try fix test
omit bead3b6 fix test
omit b24dadb fix race
omit 48e264d fix a panic
omit cedc957 polish topN close
omit 6a11951 fix test
omit d847b24 fix check
omit 1f06a4c refactor TearDown
omit 6199cef check chan Kind
omit bce5485 add comments and refactor flow
omit fc66124 add test for non-group-by TopN
omit 317899b remove batchApi
omit 72c8f74 refactor flow API
omit 1d85d7f remove timestamp flag and fix test
omit 141484a fix lint
omit ff7ffb2 refactor storage layout
omit 3ef5723 fix test
omit 4e06bfa fix format
omit 9443d64 fix imported
omit df28e0a fix measure overwrite and min and max aggregator
omit 5cb6fbf simplify AggregateFunction
omit bc04162 add comment and remove api.Row shorthand
omit 5fa8a86 fix check
omit 1fe0664 refactor storage layout
omit 323a536 add topN query
omit 72d1b6b support sort direction and json serialization
omit 1a9a45b use measureID as sharding key
omit bc57444 format
omit 9532b1b add test
omit 1a4eb2b support mapper and topN metadata reload
omit a9a5889 add measure topn
omit cf10389 add license
omit 5a825fa add batch api
omit cf17a68 refactor flow API
omit 2b8f41b add streaming api and topN aggregator
add 785e9b8 Add several metrics to measure the compression ratio (#156)
add bb433f4 Fix data race (#158)
add 122da8d This PR is mainly used to update the optimized UI (#157)
add b103a62 Use bluge to implement inverted index (#159)
new 10dbd18 add streaming api and topN aggregator
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9d1ca8f)
\
N -- N -- N refs/heads/simple-streaming-api (10dbd18)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
banyand/kv/kv.go | 3 +
banyand/liaison/grpc/stream_test.go | 2 +-
banyand/measure/encode.go | 12 +-
banyand/measure/metadata.go | 11 +-
banyand/query/processor.go | 2 +-
banyand/stream/metadata.go | 11 +-
banyand/stream/stream_write.go | 21 +-
banyand/tsdb/block.go | 23 +-
banyand/tsdb/index/writer.go | 207 +-
banyand/tsdb/indexdb.go | 45 +-
banyand/tsdb/series_write.go | 18 +-
banyand/tsdb/tsdb_test.go | 4 +-
dist/LICENSE | 33 +-
... => license-github.com-axiomhq-hyperloglog.txt} | 2 +-
...se-github.com-blevesearch-go-porterstemmer.txt} | 4 +-
... => license-github.com-blevesearch-mmap-go.txt} | 23 +-
... => license-github.com-blevesearch-segment.txt} | 0
...icense-github.com-blevesearch-snowballstem.txt} | 28 +-
...t => license-github.com-blevesearch-vellum.txt} | 0
...nse-github.com-blugelabs-bluge-segment-api.txt} | 0
....txt => license-github.com-blugelabs-bluge.txt} | 0
...txt => license-github.com-blugelabs-ice-v2.txt} | 0
...at.txt => license-github.com-blugelabs-ice.txt} | 0
....txt => license-github.com-caio-go-tdigest.txt} | 2 +-
...txt => license-github.com-dgryski-go-metro.txt} | 6 +-
.../license--babel-parser.txt} | 4 +-
.../license-csstype.txt} | 4 +-
.../license-nanoid.txt} | 2 +-
...cense-zrender.txt => license-source-map-js.txt} | 17 +-
...{license-zrender.txt => license-source-map.txt} | 17 +-
...license-vue-node-modules--vue-compiler-sfc.txt} | 0
.../license-vue-node-modules-picocolors.txt} | 2 +-
.../license-vue-node-modules-postcss.txt} | 2 +-
go.mod | 16 +-
go.sum | 60 +-
pkg/encoding/encoding.go | 6 +-
pkg/encoding/int.go | 20 +-
pkg/encoding/int_test.go | 4 +-
pkg/encoding/plain.go | 19 +-
pkg/index/index.go | 22 +-
pkg/index/inverted/field_map.go | 90 -
pkg/index/inverted/inverted.go | 329 +-
pkg/index/inverted/inverted_test.go | 73 -
pkg/index/inverted/mem.go | 241 -
pkg/index/inverted/mem_test.go | 38 -
pkg/index/inverted/term_map.go | 96 -
pkg/index/lsm/lsm.go | 17 +-
pkg/index/testcases/duration.go | 4 +-
pkg/index/testcases/service_name.go | 8 +-
pkg/logger/logger.go | 4 +
test/stress/docker-compose.yaml | 3 +
ui/LICENSE | 17 +-
ui/package-lock.json | 8343 ++++++++++----------
ui/src/assets/custom.scss | 3 +
ui/src/assets/main.scss | 6 +
ui/src/components/HeaderComponent.vue | 2 +-
.../databaseComponents/MainComponent.vue | 11 +-
.../mainComponents/DataTableComponent.vue | 2 +-
58 files changed, 4735 insertions(+), 5204 deletions(-)
copy dist/licenses/{ui-licenses/license-vue-router.txt => license-github.com-axiomhq-hyperloglog.txt} (95%)
copy dist/licenses/{license-github.com-sirupsen-logrus.txt => license-github.com-blevesearch-go-porterstemmer.txt} (92%)
copy dist/licenses/{license-github.com-magiconair-properties.txt => license-github.com-blevesearch-mmap-go.txt} (50%)
copy dist/licenses/{license-github.com-mschoch-smat.txt => license-github.com-blevesearch-segment.txt} (100%)
copy dist/licenses/{license-github.com-grpc-ecosystem-grpc-gateway-v2.txt => license-github.com-blevesearch-snowballstem.txt} (50%)
copy dist/licenses/{license-github.com-mschoch-smat.txt => license-github.com-blevesearch-vellum.txt} (100%)
copy dist/licenses/{license-github.com-mschoch-smat.txt => license-github.com-blugelabs-bluge-segment-api.txt} (100%)
copy dist/licenses/{license-github.com-mschoch-smat.txt => license-github.com-blugelabs-bluge.txt} (100%)
copy dist/licenses/{license-github.com-mschoch-smat.txt => license-github.com-blugelabs-ice-v2.txt} (100%)
copy dist/licenses/{license-github.com-mschoch-smat.txt => license-github.com-blugelabs-ice.txt} (100%)
copy dist/licenses/{license-github.com-SkyAPM-clock.txt => license-github.com-caio-go-tdigest.txt} (95%)
copy dist/licenses/{license-github.com-xiang90-probing.txt => license-github.com-dgryski-go-metro.txt} (85%)
copy dist/licenses/{license-github.com-sirupsen-logrus.txt => ui-licenses/license--babel-parser.txt} (94%)
copy dist/licenses/{license-github.com-json-iterator-go.txt => ui-licenses/license-csstype.txt} (95%)
copy dist/licenses/{license-go.etcd.io-bbolt.txt => ui-licenses/license-nanoid.txt} (95%)
copy dist/licenses/ui-licenses/{license-zrender.txt => license-source-map-js.txt} (75%)
copy dist/licenses/ui-licenses/{license-zrender.txt => license-source-map.txt} (75%)
copy dist/licenses/ui-licenses/{license-vue.txt => license-vue-node-modules--vue-compiler-sfc.txt} (100%)
copy dist/licenses/{license-github.com-davecgh-go-spew.txt => ui-licenses/license-vue-node-modules-picocolors.txt} (91%)
copy dist/licenses/{license-go.etcd.io-bbolt.txt => ui-licenses/license-vue-node-modules-postcss.txt} (95%)
delete mode 100644 pkg/index/inverted/field_map.go
delete mode 100644 pkg/index/inverted/mem.go
delete mode 100644 pkg/index/inverted/mem_test.go
delete mode 100644 pkg/index/inverted/term_map.go
[skywalking-banyandb] 01/01: add streaming api and topN aggregator
Posted by lu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lujiajing pushed a commit to branch simple-streaming-api
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 10dbd18981425ac48172793b9a8a3de02d2ae29e
Author: Megrez Lu <lu...@gmail.com>
AuthorDate: Sat Jun 18 23:18:17 2022 +0800
add streaming api and topN aggregator
---
api/data/measure.go | 6 +
banyand/liaison/grpc/measure.go | 21 +-
banyand/measure/encode.go | 2 +-
banyand/measure/field_flag_test.go | 2 +-
banyand/measure/measure.go | 35 +-
banyand/measure/measure_query.go | 35 +-
banyand/measure/measure_query_test.go | 18 +-
banyand/measure/measure_topn.go | 620 +++++++++++++++++++++
...{measure_write_test.go => measure_topn_test.go} | 92 ++-
banyand/measure/measure_write.go | 23 +-
banyand/measure/measure_write_test.go | 10 +-
banyand/measure/metadata.go | 58 +-
banyand/measure/service.go | 48 +-
banyand/metadata/metadata.go | 24 +
banyand/metadata/schema/schema.go | 7 +
banyand/query/processor.go | 13 +-
banyand/query/processor_test.go | 32 +-
banyand/query/processor_topn.go | 298 ++++++++++
banyand/query/processor_topn_test.go | 165 ++++++
banyand/query/query.go | 7 +
.../query/testdata/service_cpm_minute_data.json | 182 ++++++
banyand/stream/stream_query_test.go | 10 +-
banyand/stream/stream_write_test.go | 8 +-
banyand/tsdb/block.go | 6 +-
banyand/tsdb/scope.go | 2 +-
banyand/tsdb/segment.go | 6 +-
banyand/tsdb/series.go | 43 +-
banyand/tsdb/series_filter.go | 278 +++++++++
banyand/tsdb/series_filter_test.go | 315 +++++++++++
banyand/tsdb/series_seek.go | 52 +-
banyand/tsdb/series_seek_filter.go | 11 +-
banyand/tsdb/series_seek_sort.go | 59 +-
banyand/tsdb/series_write.go | 4 +-
banyand/tsdb/seriesdb.go | 25 +-
banyand/tsdb/seriesdb_test.go | 2 +-
dist/LICENSE | 8 +-
.../licenses/license-github.com-emirpasic-gods.txt | 41 ++
go.mod | 3 +-
go.sum | 5 +-
pkg/flow/dedup_priority_queue.go | 136 +++++
api/data/measure.go => pkg/flow/op.go | 32 +-
pkg/flow/streaming/flow.go | 120 ++++
pkg/flow/streaming/flow_sliding_window.go | 334 +++++++++++
pkg/flow/streaming/flow_sliding_window_test.go | 137 +++++
pkg/flow/streaming/flow_test.go | 186 +++++++
pkg/flow/streaming/flow_topn.go | 171 ++++++
pkg/flow/streaming/flow_topn_test.go | 94 ++++
pkg/flow/streaming/flow_unary.go | 109 ++++
pkg/flow/streaming/sink/slice.go | 82 +++
pkg/flow/streaming/sources/channel.go | 93 ++++
pkg/flow/streaming/sources/channel_test.go | 72 +++
.../flow/streaming/streaming_suite_test.go | 22 +-
pkg/flow/types.go | 174 ++++++
api/data/measure.go => pkg/flow/utils.go | 36 +-
pkg/iter/iter.go | 109 ++++
pkg/pb/v1/query.go | 108 ++++
pkg/query/aggregation/aggregation.go | 11 +-
pkg/query/aggregation/function.go | 1 +
pkg/query/logical/common.go | 6 +-
pkg/query/logical/expr.go | 4 +
pkg/query/logical/iter.go | 17 +-
pkg/query/logical/measure_plan_aggregation.go | 1 +
pkg/query/logical/measure_plan_indexscan_local.go | 49 +-
pkg/query/logical/stream_plan_indexscan_local.go | 8 +-
pkg/schema/metadata.go | 32 +-
pkg/test/flow/slice.go | 85 +++
pkg/test/flow/slice_test.go | 63 +++
pkg/test/measure/etcd.go | 43 +-
.../service_cpm_minute_nogroup_top100.json | 13 +
.../service_cpm_minute_top100.json | 16 +
pkg/timestamp/duration.go | 1 +
pkg/timestamp/nano.go | 1 +
pkg/timestamp/range.go | 1 +
73 files changed, 4612 insertions(+), 331 deletions(-)
diff --git a/api/data/measure.go b/api/data/measure.go
index a582924..08b3f1b 100644
--- a/api/data/measure.go
+++ b/api/data/measure.go
@@ -34,3 +34,9 @@ var MeasureQueryKindVersion = common.KindVersion{
Kind: "measure-query",
}
var TopicMeasureQuery = bus.BiTopic(MeasureQueryKindVersion.String())
+
+var TopNQueryKindVersion = common.KindVersion{
+ Version: "v1",
+ Kind: "topN-query",
+}
+var TopicTopNQuery = bus.BiTopic(TopNQueryKindVersion.String())
diff --git a/banyand/liaison/grpc/measure.go b/banyand/liaison/grpc/measure.go
index c77a4b5..434313a 100644
--- a/banyand/liaison/grpc/measure.go
+++ b/banyand/liaison/grpc/measure.go
@@ -106,4 +106,23 @@ func (ms *measureService) Query(_ context.Context, entityCriteria *measurev1.Que
return &measurev1.QueryResponse{DataPoints: queryMsg}, nil
}
-// TODO: implement topN
+func (ms *measureService) TopN(_ context.Context, topNRequest *measurev1.TopNRequest) (*measurev1.TopNResponse, error) {
+ if err := timestamp.CheckTimeRange(topNRequest.GetTimeRange()); err != nil {
+ return nil, status.Errorf(codes.InvalidArgument, "%v is invalid :%s", topNRequest.GetTimeRange(), err)
+ }
+
+ message := bus.NewMessage(bus.MessageID(time.Now().UnixNano()), topNRequest)
+ feat, errQuery := ms.pipeline.Publish(data.TopicTopNQuery, message)
+ if errQuery != nil {
+ return nil, errQuery
+ }
+ msg, errFeat := feat.Get()
+ if errFeat != nil {
+ return nil, errFeat
+ }
+ queryResp, ok := msg.Data().([]*measurev1.TopNList)
+ if !ok {
+ return nil, ErrQueryMsg
+ }
+ return &measurev1.TopNResponse{Lists: queryResp}, nil
+}
diff --git a/banyand/measure/encode.go b/banyand/measure/encode.go
index e1042e6..c735173 100644
--- a/banyand/measure/encode.go
+++ b/banyand/measure/encode.go
@@ -101,7 +101,7 @@ func (p *decoderPool) Put(decoder encoding.SeriesDecoder) {
const fieldFlagLength = 9
-func encoderFieldFlag(fieldSpec *databasev1.FieldSpec, interval time.Duration) []byte {
+func EncoderFieldFlag(fieldSpec *databasev1.FieldSpec, interval time.Duration) []byte {
encodingMethod := byte(fieldSpec.GetEncodingMethod().Number())
compressionMethod := byte(fieldSpec.GetCompressionMethod().Number())
bb := make([]byte, fieldFlagLength)
diff --git a/banyand/measure/field_flag_test.go b/banyand/measure/field_flag_test.go
index fba88db..98b5106 100644
--- a/banyand/measure/field_flag_test.go
+++ b/banyand/measure/field_flag_test.go
@@ -26,7 +26,7 @@ import (
)
func TestEncodeFieldFlag(t *testing.T) {
- flag := encoderFieldFlag(&databasev1.FieldSpec{
+ flag := EncoderFieldFlag(&databasev1.FieldSpec{
EncodingMethod: databasev1.EncodingMethod_ENCODING_METHOD_GORILLA,
CompressionMethod: databasev1.CompressionMethod_COMPRESSION_METHOD_ZSTD,
}, time.Minute)
diff --git a/banyand/measure/measure.go b/banyand/measure/measure.go
index 0753b2b..5cc5f1d 100644
--- a/banyand/measure/measure.go
+++ b/banyand/measure/measure.go
@@ -21,6 +21,8 @@ import (
"context"
"time"
+ "go.uber.org/multierr"
+
commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
"github.com/apache/skywalking-banyandb/banyand/tsdb"
@@ -44,11 +46,12 @@ type measure struct {
schema *databasev1.Measure
// maxObservedModRevision is the max observed revision of index rules in the spec
maxObservedModRevision int64
- db tsdb.Supplier
+ databaseSupplier tsdb.Supplier
entityLocator partition.EntityLocator
indexRules []*databasev1.IndexRule
indexWriter *index.Writer
interval time.Duration
+ processorManager *topNProcessorManager
}
func (s *measure) GetSchema() *databasev1.Measure {
@@ -72,7 +75,7 @@ func (s *measure) EntityLocator() partition.EntityLocator {
}
func (s *measure) Close() error {
- return s.indexWriter.Close()
+ return multierr.Combine(s.processorManager.Close(), s.indexWriter.Close())
}
func (s *measure) parseSpec() (err error) {
@@ -86,28 +89,42 @@ func (s *measure) parseSpec() (err error) {
}
type measureSpec struct {
- schema *databasev1.Measure
- indexRules []*databasev1.IndexRule
+ schema *databasev1.Measure
+ indexRules []*databasev1.IndexRule
+ topNAggregations []*databasev1.TopNAggregation
}
func openMeasure(shardNum uint32, db tsdb.Supplier, spec measureSpec, l *logger.Logger) (*measure, error) {
- sm := &measure{
+ m := &measure{
shardNum: shardNum,
schema: spec.schema,
indexRules: spec.indexRules,
l: l,
}
- if err := sm.parseSpec(); err != nil {
+ if err := m.parseSpec(); err != nil {
return nil, err
}
ctx := context.WithValue(context.Background(), logger.ContextKey, l)
- sm.db = db
- sm.indexWriter = index.NewWriter(ctx, index.WriterOptions{
+ m.databaseSupplier = db
+ m.indexWriter = index.NewWriter(ctx, index.WriterOptions{
DB: db,
ShardNum: shardNum,
Families: spec.schema.TagFamilies,
IndexRules: spec.indexRules,
})
- return sm, nil
+
+ m.processorManager = &topNProcessorManager{
+ l: l,
+ m: m,
+ topNSchemas: spec.topNAggregations,
+ processorMap: make(map[*commonv1.Metadata]*topNStreamingProcessor),
+ }
+
+ err := m.processorManager.start()
+ if err != nil {
+ return nil, err
+ }
+
+ return m, nil
}
diff --git a/banyand/measure/measure_query.go b/banyand/measure/measure_query.go
index d47f723..46f62b8 100644
--- a/banyand/measure/measure_query.go
+++ b/banyand/measure/measure_query.go
@@ -19,6 +19,7 @@ package measure
import (
"io"
+ "time"
"github.com/pkg/errors"
"google.golang.org/protobuf/proto"
@@ -44,15 +45,21 @@ type Measure interface {
io.Closer
Write(value *measurev1.DataPointValue) error
Shards(entity tsdb.Entity) ([]tsdb.Shard, error)
+ CompanionShards(metadata *commonv1.Metadata) ([]tsdb.Shard, error)
Shard(id common.ShardID) (tsdb.Shard, error)
ParseTagFamily(family string, item tsdb.Item) (*modelv1.TagFamily, error)
ParseField(name string, item tsdb.Item) (*measurev1.DataPoint_Field, error)
GetSchema() *databasev1.Measure
GetIndexRules() []*databasev1.IndexRule
+ GetInterval() time.Duration
}
var _ Measure = (*measure)(nil)
+func (s *measure) GetInterval() time.Duration {
+ return s.interval
+}
+
func (s *measure) Shards(entity tsdb.Entity) ([]tsdb.Shard, error) {
wrap := func(shards []tsdb.Shard) []tsdb.Shard {
result := make([]tsdb.Shard, len(shards))
@@ -61,7 +68,7 @@ func (s *measure) Shards(entity tsdb.Entity) ([]tsdb.Shard, error) {
}
return result
}
- db := s.db.SupplyTSDB()
+ db := s.databaseSupplier.SupplyTSDB()
if len(entity) < 1 {
return wrap(db.Shards()), nil
}
@@ -74,15 +81,31 @@ func (s *measure) Shards(entity tsdb.Entity) ([]tsdb.Shard, error) {
if err != nil {
return nil, err
}
- shard, err := db.Shard(common.ShardID(shardID))
+ shard, err := s.Shard(common.ShardID(shardID))
if err != nil {
return nil, err
}
- return []tsdb.Shard{tsdb.NewScopedShard(tsdb.Entry(s.name), shard)}, nil
+ return []tsdb.Shard{shard}, nil
+}
+
+func (s *measure) CompanionShards(metadata *commonv1.Metadata) ([]tsdb.Shard, error) {
+ wrap := func(shards []tsdb.Shard) []tsdb.Shard {
+ result := make([]tsdb.Shard, len(shards))
+ for i := 0; i < len(shards); i++ {
+ result[i] = tsdb.NewScopedShard(tsdb.Entry(formatMeasureCompanionPrefix(s.name, metadata.GetName())), shards[i])
+ }
+ return result
+ }
+ db := s.databaseSupplier.SupplyTSDB()
+ return wrap(db.Shards()), nil
+}
+
+func formatMeasureCompanionPrefix(measureName, name string) string {
+ return measureName + "." + name
}
func (s *measure) Shard(id common.ShardID) (tsdb.Shard, error) {
- shard, err := s.db.SupplyTSDB().Shard(id)
+ shard, err := s.databaseSupplier.SupplyTSDB().Shard(id)
if err != nil {
return nil, err
}
@@ -131,11 +154,11 @@ func (s *measure) ParseField(name string, item tsdb.Item) (*measurev1.DataPoint_
break
}
}
- bytes, err := item.Family(familyIdentity(name, encoderFieldFlag(fieldSpec, s.interval)))
+ bytes, err := item.Family(familyIdentity(name, EncoderFieldFlag(fieldSpec, s.interval)))
if err != nil {
return nil, err
}
- fieldValue := decodeFieldValue(bytes, fieldSpec)
+ fieldValue := DecodeFieldValue(bytes, fieldSpec)
return &measurev1.DataPoint_Field{
Name: name,
Value: fieldValue,
diff --git a/banyand/measure/measure_query_test.go b/banyand/measure/measure_query_test.go
index b4bcb40..a766e91 100644
--- a/banyand/measure/measure_query_test.go
+++ b/banyand/measure/measure_query_test.go
@@ -82,12 +82,15 @@ var _ = Describe("Query service_cpm_minute", func() {
iter, err := seeker.Seek()
Expect(err).ShouldNot(HaveOccurred())
Expect(len(iter)).To(Equal(1))
- defer func(iterator tsdb.Iterator) {
+ defer func(iterator tsdb.ItemIterator) {
Expect(iterator.Close()).ShouldNot(HaveOccurred())
}(iter[0])
i := 0
- for ; iter[0].Next(); i++ {
- item := iter[0].Val()
+ for {
+ item, hasNext := iter[0].Next()
+ if !hasNext {
+ break
+ }
tagFamily, err := measure.ParseTagFamily("default", item)
Expect(err).ShouldNot(HaveOccurred())
Expect(len(tagFamily.Tags)).To(Equal(2))
@@ -97,6 +100,7 @@ var _ = Describe("Query service_cpm_minute", func() {
value, err := measure.ParseField("value", item)
Expect(err).ShouldNot(HaveOccurred())
Expect(value.GetValue().GetInt().Value).To(Equal(expectedFields[i][1]))
+ i++
}
Expect(len(expectedFields)).To(Equal(i))
}
@@ -200,13 +204,13 @@ var _ = Describe("Query service_traffic", func() {
iter, err := seeker.Seek()
Expect(err).ShouldNot(HaveOccurred())
Expect(len(iter)).To(Equal(1))
- defer func(iterator tsdb.Iterator) {
+ defer func(iterator tsdb.ItemIterator) {
_ = iterator.Close()
}(iter[0])
- if !iter[0].Next() {
+ item, hasNext := iter[0].Next()
+ if !hasNext {
continue
}
- item := iter[0].Val()
tagFamily, err := measure.ParseTagFamily("default", item)
Expect(err).ShouldNot(HaveOccurred())
Expect(len(tagFamily.Tags)).To(Equal(6))
@@ -220,7 +224,7 @@ var _ = Describe("Query service_traffic", func() {
if !existServiceID {
Fail("doesn't get service_id")
}
- if iter[0].Next() {
+ if _, hasNext = iter[0].Next(); hasNext {
Fail("should only one data point in this series")
}
}
diff --git a/banyand/measure/measure_topn.go b/banyand/measure/measure_topn.go
new file mode 100644
index 0000000..9c664b0
--- /dev/null
+++ b/banyand/measure/measure_topn.go
@@ -0,0 +1,620 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "encoding/base64"
+ "io"
+ "strconv"
+ "strings"
+ "sync"
+ "time"
+
+ "github.com/pkg/errors"
+ "go.uber.org/multierr"
+ "golang.org/x/exp/slices"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/banyand/tsdb"
+ "github.com/apache/skywalking-banyandb/pkg/bus"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+ "github.com/apache/skywalking-banyandb/pkg/flow/streaming"
+ "github.com/apache/skywalking-banyandb/pkg/flow/streaming/sources"
+ "github.com/apache/skywalking-banyandb/pkg/logger"
+ "github.com/apache/skywalking-banyandb/pkg/partition"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/timestamp"
+)
+
+const (
+ timeBucketFormat = "200601021504"
+ TopNTagFamily = "__topN__"
+)
+
+var (
+ _ bus.MessageListener = (*topNProcessCallback)(nil)
+ _ io.Closer = (*topNStreamingProcessor)(nil)
+ _ io.Closer = (*topNProcessorManager)(nil)
+ _ flow.Sink = (*topNStreamingProcessor)(nil)
+
+ TopNValueFieldSpec = &databasev1.FieldSpec{
+ Name: "value",
+ FieldType: databasev1.FieldType_FIELD_TYPE_INT,
+ EncodingMethod: databasev1.EncodingMethod_ENCODING_METHOD_GORILLA,
+ CompressionMethod: databasev1.CompressionMethod_COMPRESSION_METHOD_ZSTD,
+ }
+)
+
+type topNStreamingProcessor struct {
+ flow.ComponentState
+ l *logger.Logger
+ shardNum uint32
+ interval time.Duration
+ slideSize time.Duration
+ topNSchema *databasev1.TopNAggregation
+ databaseSupplier tsdb.Supplier
+ src chan interface{}
+ in chan flow.StreamRecord
+ errCh <-chan error
+ stopCh chan struct{}
+ streamingFlow flow.Flow
+}
+
+func (t *topNStreamingProcessor) In() chan<- flow.StreamRecord {
+ return t.in
+}
+
+func (t *topNStreamingProcessor) Setup(ctx context.Context) error {
+ t.Add(1)
+ go t.run(ctx)
+ return nil
+}
+
+func (t *topNStreamingProcessor) run(ctx context.Context) {
+ defer t.Done()
+ for {
+ select {
+ case record, ok := <-t.in:
+ if !ok {
+ return
+ }
+ if err := t.writeStreamRecord(record); err != nil {
+ t.l.Err(err).Msg("fail to write stream record")
+ }
+ case <-ctx.Done():
+ return
+ }
+ }
+}
+
+// Teardown is called by the Flow as a lifecycle hook.
+// So we should not block on err channel within this method.
+func (t *topNStreamingProcessor) Teardown(ctx context.Context) error {
+ t.Wait()
+ return nil
+}
+
+func (t *topNStreamingProcessor) Close() error {
+ close(t.src)
+ // close streaming flow
+ err := t.streamingFlow.Close()
+ // and wait for error channel close
+ <-t.stopCh
+ t.stopCh = nil
+ return err
+}
+
+func (t *topNStreamingProcessor) writeStreamRecord(record flow.StreamRecord) error {
+ tuples, ok := record.Data().([]*streaming.Tuple2)
+ if !ok {
+ return errors.New("invalid data type")
+ }
+ // eventTime is the start time of a timeWindow
+ eventTime := time.UnixMilli(record.TimestampMillis())
+ // down-sampling to a time bucket as measure ID
+ timeBucket := t.downSampleTimeBucket(eventTime)
+ var err error
+ for rankNum, tuple := range tuples {
+ fieldValue := tuple.V1.(int64)
+ data := tuple.V2.(flow.StreamRecord).Data().(flow.Data)
+ err = multierr.Append(err, t.writeData(eventTime, timeBucket, fieldValue, data, rankNum))
+ }
+ return err
+}
+
+func (t *topNStreamingProcessor) writeData(eventTime time.Time, timeBucket string, fieldValue int64, data flow.Data, rankNum int) error {
+ var tagValues []*modelv1.TagValue
+ if len(t.topNSchema.GetGroupByTagNames()) > 0 {
+ var ok bool
+ if tagValues, ok = data[2].([]*modelv1.TagValue); !ok {
+ return errors.New("fail to extract tag values from topN result")
+ }
+ }
+ entity, shardID, err := t.locate(tagValues, rankNum)
+ if err != nil {
+ return err
+ }
+ shard, err := t.databaseSupplier.SupplyTSDB().Shard(shardID)
+ if err != nil {
+ return err
+ }
+ series, err := shard.Series().GetByHashKey(tsdb.HashEntity(entity))
+ if err != nil {
+ return err
+ }
+ span, err := series.Span(timestamp.NewInclusiveTimeRangeDuration(eventTime, 0))
+ if err != nil {
+ if span != nil {
+ _ = span.Close()
+ }
+ return err
+ }
+ // measureID is consist of three parts,
+ // 1. groupValues
+ // 2. rankNumber
+ // 3. timeBucket
+ measureID := data[0].(string) + "_" + strconv.Itoa(rankNum) + "_" + timeBucket
+ writeFn := func() (tsdb.Writer, error) {
+ builder := span.WriterBuilder().Time(eventTime)
+ virtualTagFamily := &modelv1.TagFamilyForWrite{
+ Tags: []*modelv1.TagValue{
+ // MeasureID
+ {
+ Value: &modelv1.TagValue_Id{
+ Id: &modelv1.ID{
+ Value: measureID,
+ },
+ },
+ },
+ // GroupValues for merge in post processor
+ {
+ Value: &modelv1.TagValue_Str{
+ Str: &modelv1.Str{
+ Value: data[0].(string),
+ },
+ },
+ },
+ },
+ }
+ payload, errMarshal := proto.Marshal(virtualTagFamily)
+ if errMarshal != nil {
+ return nil, errMarshal
+ }
+ builder.Family(familyIdentity(TopNTagFamily, TagFlag), payload)
+ virtualFieldValue := &modelv1.FieldValue{
+ Value: &modelv1.FieldValue_Int{
+ Int: &modelv1.Int{
+ Value: fieldValue,
+ },
+ },
+ }
+ fieldData := encodeFieldValue(virtualFieldValue)
+ builder.Family(familyIdentity(TopNValueFieldSpec.GetName(), EncoderFieldFlag(TopNValueFieldSpec, t.interval)), fieldData)
+ writer, errWrite := builder.Build()
+ if errWrite != nil {
+ return nil, errWrite
+ }
+ _, errWrite = writer.Write()
+ t.l.Debug().
+ Time("ts", eventTime).
+ Int("ts_nano", eventTime.Nanosecond()).
+ Uint64("series_id", uint64(series.ID())).
+ Uint64("item_id", uint64(writer.ItemID().ID)).
+ Int("shard_id", int(shardID)).
+ Msg("write measure")
+ return writer, errWrite
+ }
+ _, err = writeFn()
+ if err != nil {
+ _ = span.Close()
+ return err
+ }
+ return span.Close()
+}
+
+func (t *topNStreamingProcessor) downSampleTimeBucket(eventTime time.Time) string {
+ return time.UnixMilli(eventTime.UnixMilli() - eventTime.UnixMilli()%t.interval.Milliseconds()).
+ Format(timeBucketFormat)
+}
+
+func (t *topNStreamingProcessor) locate(tagValues []*modelv1.TagValue, rankNum int) (tsdb.Entity, common.ShardID, error) {
+ if len(t.topNSchema.GetGroupByTagNames()) != len(tagValues) {
+ return nil, 0, errors.New("no enough tag values for the entity")
+ }
+ entity := make(tsdb.Entity, 1+1+len(t.topNSchema.GetGroupByTagNames()))
+ // entity prefix
+ entity[0] = []byte(formatMeasureCompanionPrefix(t.topNSchema.GetSourceMeasure().GetName(),
+ t.topNSchema.GetMetadata().GetName()))
+ entity[1] = convert.Int64ToBytes(int64(rankNum))
+ // measureID as sharding key
+ for idx, tagVal := range tagValues {
+ var innerErr error
+ entity[idx+2], innerErr = pbv1.MarshalIndexFieldValue(tagVal)
+ if innerErr != nil {
+ return nil, 0, innerErr
+ }
+ }
+ id, err := partition.ShardID(entity.Marshal(), t.shardNum)
+ if err != nil {
+ return nil, 0, err
+ }
+ return entity, common.ShardID(id), nil
+}
+
+func (t *topNStreamingProcessor) start() *topNStreamingProcessor {
+ t.errCh = t.streamingFlow.Window(streaming.NewTumblingTimeWindows(t.slideSize)).
+ TopN(int(t.topNSchema.GetCountersNumber()),
+ streaming.WithSortKeyExtractor(func(record flow.StreamRecord) int64 {
+ return record.Data().(flow.Data)[1].(int64)
+ }),
+ streaming.OrderBy(t.topNSchema.GetFieldValueSort()),
+ ).To(t).Open()
+ go t.handleError()
+ return t
+}
+
+func (t *topNStreamingProcessor) handleError() {
+ for err := range t.errCh {
+ t.l.Err(err).Str("topN", t.topNSchema.GetMetadata().GetName()).
+ Msg("error occurred during flow setup or process")
+ }
+ t.stopCh <- struct{}{}
+}
+
+// topNProcessorManager manages multiple topNStreamingProcessor(s) belonging to a single measure
+type topNProcessorManager struct {
+ // RWMutex here is to protect the processorMap from data race, i.e.
+ // the send operation to the underlying channel vs. the close of the channel
+ // TODO: this can be optimized if the bus Listener can be synchronously finished,
+ sync.RWMutex
+ l *logger.Logger
+ m *measure
+ topNSchemas []*databasev1.TopNAggregation
+ processorMap map[*commonv1.Metadata]*topNStreamingProcessor
+}
+
+func (manager *topNProcessorManager) Close() error {
+ manager.Lock()
+ defer manager.Unlock()
+ var err error
+ for _, processor := range manager.processorMap {
+ err = multierr.Append(err, processor.Close())
+ }
+ return err
+}
+
+func (manager *topNProcessorManager) onMeasureWrite(request *measurev1.WriteRequest) error {
+ manager.RLock()
+ defer manager.RUnlock()
+ for _, processor := range manager.processorMap {
+ processor.src <- flow.NewStreamRecordWithTimestampPb(request.GetDataPoint(), request.GetDataPoint().GetTimestamp())
+ }
+
+ return nil
+}
+
+func (manager *topNProcessorManager) start() error {
+ interval := manager.m.interval
+ // use 40% of the data point interval as the flush interval,
+ // which means roughly the record located in the same time bucket will be persistent twice.
+ // |---------------------------------|
+ // | 40% | 40% | 20% |
+ // | flush flush |
+ // |---------------------------------|
+ slideSize := time.Duration(float64(interval.Nanoseconds()) * 0.4)
+ for _, topNSchema := range manager.topNSchemas {
+ srcCh := make(chan interface{})
+ src, _ := sources.NewChannel(srcCh)
+ streamingFlow := streaming.New(src)
+
+ if conditions := topNSchema.GetCriteria(); len(conditions) > 0 {
+ filters, buildErr := manager.buildFilters(conditions)
+ if buildErr != nil {
+ return buildErr
+ }
+ streamingFlow = streamingFlow.Filter(filters)
+ }
+
+ mapper, innerErr := manager.buildMapper(topNSchema.GetFieldName(), topNSchema.GetGroupByTagNames()...)
+ if innerErr != nil {
+ return innerErr
+ }
+ streamingFlow = streamingFlow.Map(mapper)
+
+ processor := &topNStreamingProcessor{
+ l: manager.l,
+ shardNum: manager.m.shardNum,
+ interval: interval,
+ slideSize: slideSize,
+ topNSchema: topNSchema,
+ databaseSupplier: manager.m.databaseSupplier,
+ src: srcCh,
+ in: make(chan flow.StreamRecord),
+ stopCh: make(chan struct{}),
+ streamingFlow: streamingFlow,
+ }
+
+ manager.processorMap[topNSchema.GetSourceMeasure()] = processor.start()
+ }
+
+ return nil
+}
+
+func (manager *topNProcessorManager) buildFilters(criteria []*modelv1.Criteria) (flow.UnaryFunc[bool], error) {
+ var filters []conditionFilter
+ for _, group := range criteria {
+ for _, cond := range group.GetConditions() {
+ fIdx, tIdx, spec := pbv1.FindTagByName(manager.m.GetSchema().GetTagFamilies(), cond.GetName())
+ if spec == nil {
+ return nil, errors.New("fail to parse tag by name")
+ }
+ filters = append(filters, manager.buildFilterForTag(fIdx, tIdx, cond))
+ }
+ }
+
+ if len(filters) == 0 {
+ return func(_ context.Context, value interface{}) bool {
+ return true
+ }, nil
+ }
+
+ return func(_ context.Context, dataPoint any) bool {
+ tfs := dataPoint.(*measurev1.DataPointValue).GetTagFamilies()
+ for _, f := range filters {
+ if !f.predicate(tfs) {
+ return false
+ }
+ }
+ return true
+ }, nil
+}
+
+func (manager *topNProcessorManager) buildFilterForTag(familyOffset, tagOffset int, cond *modelv1.Condition) conditionFilter {
+ switch v := cond.GetValue().GetValue().(type) {
+ case *modelv1.TagValue_Int:
+ return &int64TagFilter{
+ TagLocator: partition.TagLocator{
+ FamilyOffset: familyOffset,
+ TagOffset: tagOffset,
+ },
+ op: cond.GetOp(),
+ val: v.Int.GetValue(),
+ }
+ case *modelv1.TagValue_Str:
+ return &strTagFilter{
+ TagLocator: partition.TagLocator{
+ FamilyOffset: familyOffset,
+ TagOffset: tagOffset,
+ },
+ op: cond.GetOp(),
+ val: v.Str.GetValue(),
+ }
+ case *modelv1.TagValue_Id:
+ return &idTagFilter{
+ TagLocator: partition.TagLocator{
+ FamilyOffset: familyOffset,
+ TagOffset: tagOffset,
+ },
+ op: cond.GetOp(),
+ val: v.Id.GetValue(),
+ }
+ default:
+ return nil
+ }
+}
+
+func (manager *topNProcessorManager) buildMapper(fieldName string, groupByNames ...string) (flow.UnaryFunc[any], error) {
+ fieldIdx := slices.IndexFunc(manager.m.GetSchema().GetFields(), func(spec *databasev1.FieldSpec) bool {
+ return spec.GetName() == fieldName
+ })
+ if fieldIdx == -1 {
+ return nil, errors.New("invalid fieldName")
+ }
+ if len(groupByNames) == 0 {
+ return func(_ context.Context, request any) any {
+ dataPoint := request.(*measurev1.DataPointValue)
+ return flow.Data{
+ // save string representation of group values as the key, i.e. v1
+ "",
+ // field value as v2
+ // TODO: we only support int64
+ dataPoint.GetFields()[fieldIdx].GetInt().GetValue(),
+ // groupBy tag values as v3
+ nil,
+ }
+ }, nil
+ }
+ groupLocator, err := newGroupLocator(manager.m.GetSchema(), groupByNames)
+ if err != nil {
+ return nil, err
+ }
+ return func(_ context.Context, request any) any {
+ dataPoint := request.(*measurev1.DataPointValue)
+ return flow.Data{
+ // save string representation of group values as the key, i.e. v1
+ strings.Join(transform(groupLocator, func(locator partition.TagLocator) string {
+ return stringify(dataPoint.GetTagFamilies()[locator.FamilyOffset].GetTags()[locator.TagOffset])
+ }), "|"),
+ // field value as v2
+ // TODO: we only support int64
+ dataPoint.GetFields()[fieldIdx].GetInt().GetValue(),
+ // groupBy tag values as v3
+ transform(groupLocator, func(locator partition.TagLocator) *modelv1.TagValue {
+ return dataPoint.GetTagFamilies()[locator.FamilyOffset].GetTags()[locator.TagOffset]
+ }),
+ }
+ }, nil
+}
+
+// topNProcessCallback listens pipeline for writing requests
+type topNProcessCallback struct {
+ l *logger.Logger
+ schemaRepo *schemaRepo
+}
+
+func setUpStreamingProcessCallback(l *logger.Logger, schemaRepo *schemaRepo) bus.MessageListener {
+ return &topNProcessCallback{
+ l: l,
+ schemaRepo: schemaRepo,
+ }
+}
+
+func (cb *topNProcessCallback) Rev(message bus.Message) (resp bus.Message) {
+ writeEvent, ok := message.Data().(*measurev1.InternalWriteRequest)
+ if !ok {
+ cb.l.Warn().Msg("invalid event data type")
+ return
+ }
+
+ // first get measure existence
+ m, ok := cb.schemaRepo.loadMeasure(writeEvent.GetRequest().GetMetadata())
+ if !ok {
+ cb.l.Warn().Msg("cannot find measure definition")
+ return
+ }
+
+ err := m.processorManager.onMeasureWrite(writeEvent.GetRequest())
+ if err != nil {
+ cb.l.Debug().Err(err).Msg("fail to send to the streaming processor")
+ }
+
+ return
+}
+
+var (
+ _ conditionFilter = (*strTagFilter)(nil)
+ _ conditionFilter = (*int64TagFilter)(nil)
+)
+
+type conditionFilter interface {
+ predicate(tagFamilies []*modelv1.TagFamilyForWrite) bool
+}
+
+type strTagFilter struct {
+ partition.TagLocator
+ op modelv1.Condition_BinaryOp
+ val string
+}
+
+func (f *strTagFilter) predicate(tagFamilies []*modelv1.TagFamilyForWrite) bool {
+ strValue := tagFamilies[f.FamilyOffset].GetTags()[f.TagOffset].GetStr().GetValue()
+ switch f.op {
+ case modelv1.Condition_BINARY_OP_EQ:
+ return strValue == f.val
+ case modelv1.Condition_BINARY_OP_NE:
+ return strValue != f.val
+ }
+ return false
+}
+
+type idTagFilter struct {
+ partition.TagLocator
+ op modelv1.Condition_BinaryOp
+ val string
+}
+
+func (f *idTagFilter) predicate(tagFamilies []*modelv1.TagFamilyForWrite) bool {
+ val := tagFamilies[f.FamilyOffset].GetTags()[f.TagOffset].GetId().GetValue()
+ switch f.op {
+ case modelv1.Condition_BINARY_OP_EQ:
+ return val == f.val
+ case modelv1.Condition_BINARY_OP_NE:
+ return val != f.val
+ }
+ return false
+}
+
+type int64TagFilter struct {
+ partition.TagLocator
+ op modelv1.Condition_BinaryOp
+ val int64
+}
+
+func (f *int64TagFilter) predicate(tagFamilies []*modelv1.TagFamilyForWrite) bool {
+ val := tagFamilies[f.FamilyOffset].GetTags()[f.TagOffset].GetInt().GetValue()
+ switch f.op {
+ case modelv1.Condition_BINARY_OP_EQ:
+ return val == f.val
+ case modelv1.Condition_BINARY_OP_NE:
+ return val != f.val
+ case modelv1.Condition_BINARY_OP_GE:
+ return val >= f.val
+ case modelv1.Condition_BINARY_OP_GT:
+ return val > f.val
+ case modelv1.Condition_BINARY_OP_LE:
+ return val <= f.val
+ case modelv1.Condition_BINARY_OP_LT:
+ return val < f.val
+ }
+ return false
+}
+
+// groupTagsLocator can be used to locate tags within families
+type groupTagsLocator []partition.TagLocator
+
+// newGroupLocator generates a groupTagsLocator which strictly preserve the order of groupByNames
+func newGroupLocator(m *databasev1.Measure, groupByNames []string) (groupTagsLocator, error) {
+ groupTags := make([]partition.TagLocator, 0, len(groupByNames))
+ for _, groupByName := range groupByNames {
+ fIdx, tIdx, spec := pbv1.FindTagByName(m.GetTagFamilies(), groupByName)
+ if spec == nil {
+ return nil, errors.New("tag is not found")
+ }
+ groupTags = append(groupTags, partition.TagLocator{
+ FamilyOffset: fIdx,
+ TagOffset: tIdx,
+ })
+ }
+ return groupTags, nil
+}
+
+func stringify(tagValue *modelv1.TagValue) string {
+ switch v := tagValue.GetValue().(type) {
+ case *modelv1.TagValue_Str:
+ return v.Str.GetValue()
+ case *modelv1.TagValue_Id:
+ return v.Id.GetValue()
+ case *modelv1.TagValue_Int:
+ return strconv.FormatInt(v.Int.GetValue(), 10)
+ case *modelv1.TagValue_BinaryData:
+ return base64.StdEncoding.EncodeToString(v.BinaryData)
+ case *modelv1.TagValue_IntArray:
+ return strings.Join(transform(v.IntArray.GetValue(), func(num int64) string {
+ return strconv.FormatInt(num, 10)
+ }), ",")
+ case *modelv1.TagValue_StrArray:
+ return strings.Join(v.StrArray.GetValue(), ",")
+ default:
+ return "<nil>"
+ }
+}
+
+func transform[I, O any](input []I, mapper func(I) O) []O {
+ output := make([]O, len(input))
+ for i := range input {
+ output[i] = mapper(input[i])
+ }
+ return output
+}
diff --git a/banyand/measure/measure_write_test.go b/banyand/measure/measure_topn_test.go
similarity index 54%
copy from banyand/measure/measure_write_test.go
copy to banyand/measure/measure_topn_test.go
index 935de8d..2fd187d 100644
--- a/banyand/measure/measure_write_test.go
+++ b/banyand/measure/measure_topn_test.go
@@ -18,35 +18,37 @@
package measure_test
import (
- "embed"
- "encoding/json"
+ "context"
"time"
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
- "google.golang.org/protobuf/encoding/protojson"
- "google.golang.org/protobuf/types/known/timestamppb"
commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
- measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
"github.com/apache/skywalking-banyandb/banyand/measure"
"github.com/apache/skywalking-banyandb/banyand/tsdb"
"github.com/apache/skywalking-banyandb/pkg/timestamp"
)
-var _ = Describe("Write and Update service_cpm_minute", func() {
+var _ = Describe("Write and Count TopN for service_cpm_minute", func() {
var svcs *services
var deferFn func()
var measure measure.Measure
var baseTime time.Time
- count := func() (num int) {
+ count := func(topNMetadata *commonv1.Metadata) (num int) {
// Retrieve all shards
- shards, err := measure.Shards(nil)
+ shards, err := measure.CompanionShards(topNMetadata)
Expect(err).ShouldNot(HaveOccurred())
+ topNSchema, err := svcs.metadataService.TopNAggregationRegistry().GetTopNAggregation(context.TODO(), topNMetadata)
+ Expect(err).ShouldNot(HaveOccurred())
+ entity := make(tsdb.Entity, 1+len(topNSchema.GetGroupByTagNames()))
+ for i := range entity {
+ entity[i] = tsdb.AnyEntry
+ }
for _, shard := range shards {
- sl, err := shard.Series().List(tsdb.NewPath([]tsdb.Entry{tsdb.AnyEntry}))
+ sl, err := shard.Series().List(tsdb.NewPath(entity))
Expect(err).ShouldNot(HaveOccurred())
for _, series := range sl {
seriesSpan, err := series.Span(timestamp.NewInclusiveTimeRangeDuration(baseTime, 1*time.Hour))
@@ -59,11 +61,15 @@ var _ = Describe("Write and Update service_cpm_minute", func() {
iters, err := seeker.Seek()
Expect(err).ShouldNot(HaveOccurred())
for _, iter := range iters {
- defer func(iterator tsdb.Iterator) {
+ defer func(iterator tsdb.ItemIterator) {
Expect(iterator.Close()).ShouldNot(HaveOccurred())
}(iter)
- for iter.Next() {
- num++
+ for {
+ if _, hasNext := iter.Next(); hasNext {
+ num++
+ } else {
+ break
+ }
}
}
}
@@ -83,58 +89,30 @@ var _ = Describe("Write and Update service_cpm_minute", func() {
AfterEach(func() {
deferFn()
})
- DescribeTable("writes", func(metadata *commonv1.Metadata, dataFile string, expectDatapointsNum int) {
+ DescribeTable("writes", func(measureMetadata *commonv1.Metadata, topNMetadata *commonv1.Metadata,
+ dataTextualFile string, expectDataPointsNum int,
+ ) {
var err error
- measure, err = svcs.measure.Measure(metadata)
+ measure, err = svcs.measure.Measure(measureMetadata)
Expect(err).ShouldNot(HaveOccurred())
- baseTime = writeData(dataFile, measure)
- Expect(count()).To(Equal(expectDatapointsNum))
+ baseTime = writeData(dataTextualFile, measure)
+ Eventually(func(g Gomega) {
+ g.Expect(count(topNMetadata)).To(Equal(expectDataPointsNum))
+ }).WithTimeout(5 * time.Second).WithPolling(1 * time.Second).Should(Succeed())
},
Entry("service_cpm_minute", &commonv1.Metadata{
Name: "service_cpm_minute",
Group: "sw_metric",
- }, "service_cpm_minute_data.json", 3),
- Entry("service_traffic", &commonv1.Metadata{
- Name: "service_traffic",
+ }, &commonv1.Metadata{
+ Name: "service_cpm_minute_top100",
Group: "sw_metric",
- }, "service_traffic_data.json", 3),
- )
- It("updates", func() {
- metadata := &commonv1.Metadata{
+ }, "service_cpm_minute_data.json", 2),
+ Entry("service_cpm_minute without groupBy", &commonv1.Metadata{
Name: "service_cpm_minute",
Group: "sw_metric",
- }
- var err error
- measure, err = svcs.measure.Measure(metadata)
- Expect(err).ShouldNot(HaveOccurred())
- baseTime = writeData("service_cpm_minute_data.json", measure)
- Expect(count()).To(Equal(3))
- writeDataWithBaseTime(baseTime, "service_cpm_minute_data1.json", measure)
- Expect(count()).To(Equal(3))
- })
+ }, &commonv1.Metadata{
+ Name: "service_cpm_minute_no_group_by_top100",
+ Group: "sw_metric",
+ }, "service_cpm_minute_data.json", 2),
+ )
})
-
-//go:embed testdata/*.json
-var dataFS embed.FS
-
-func writeData(dataFile string, measure measure.Measure) (baseTime time.Time) {
- baseTime = timestamp.NowMilli()
- writeDataWithBaseTime(baseTime, dataFile, measure)
- return baseTime
-}
-
-func writeDataWithBaseTime(baseTime time.Time, dataFile string, measure measure.Measure) time.Time {
- var templates []interface{}
- content, err := dataFS.ReadFile("testdata/" + dataFile)
- Expect(err).ShouldNot(HaveOccurred())
- Expect(json.Unmarshal(content, &templates)).ShouldNot(HaveOccurred())
- for i, template := range templates {
- rawDataPointValue, errMarshal := json.Marshal(template)
- Expect(errMarshal).ShouldNot(HaveOccurred())
- dataPointValue := &measurev1.DataPointValue{}
- Expect(protojson.Unmarshal(rawDataPointValue, dataPointValue)).ShouldNot(HaveOccurred())
- dataPointValue.Timestamp = timestamppb.New(baseTime.Add(time.Duration(i) * time.Minute))
- Expect(measure.Write(dataPointValue)).Should(Succeed())
- }
- return baseTime
-}
diff --git a/banyand/measure/measure_write.go b/banyand/measure/measure_write.go
index 58d0bab..af7073c 100644
--- a/banyand/measure/measure_write.go
+++ b/banyand/measure/measure_write.go
@@ -40,9 +40,10 @@ var (
ErrMalformedElement = errors.New("element is malformed")
ErrMalformedFieldFlag = errors.New("field flag is malformed")
- TagFlag []byte = make([]byte, fieldFlagLength)
+ TagFlag = make([]byte, fieldFlagLength)
)
+// Write is for testing
func (s *measure) Write(value *measurev1.DataPointValue) error {
entity, shardID, err := s.entityLocator.Locate(s.name, value.GetTagFamilies(), s.shardNum)
if err != nil {
@@ -56,6 +57,15 @@ func (s *measure) Write(value *measurev1.DataPointValue) error {
close(waitCh)
return err
}
+ // send to stream processor
+ err = s.processorManager.onMeasureWrite(&measurev1.WriteRequest{
+ Metadata: s.GetMetadata(),
+ DataPoint: value,
+ })
+ if err != nil {
+ close(waitCh)
+ return err
+ }
<-waitCh
return nil
}
@@ -73,7 +83,7 @@ func (s *measure) write(shardID common.ShardID, seriesHashKey []byte, value *mea
if fLen > len(sm.TagFamilies) {
return errors.Wrap(ErrMalformedElement, "tag family number is more than expected")
}
- shard, err := s.db.SupplyTSDB().Shard(shardID)
+ shard, err := s.databaseSupplier.SupplyTSDB().Shard(shardID)
if err != nil {
return err
}
@@ -127,7 +137,7 @@ func (s *measure) write(shardID common.ShardID, seriesHashKey []byte, value *mea
if data == nil {
continue
}
- builder.Family(familyIdentity(sm.GetFields()[fi].GetName(), encoderFieldFlag(fieldSpec, s.interval)), data)
+ builder.Family(familyIdentity(sm.GetFields()[fi].GetName(), EncoderFieldFlag(fieldSpec, s.interval)), data)
}
writer, errWrite := builder.Build()
if errWrite != nil {
@@ -167,12 +177,11 @@ type writeCallback struct {
schemaRepo *schemaRepo
}
-func setUpWriteCallback(l *logger.Logger, schemaRepo *schemaRepo) *writeCallback {
- wcb := &writeCallback{
+func setUpWriteCallback(l *logger.Logger, schemaRepo *schemaRepo) bus.MessageListener {
+ return &writeCallback{
l: l,
schemaRepo: schemaRepo,
}
- return wcb
}
func (w *writeCallback) Rev(message bus.Message) (resp bus.Message) {
@@ -210,7 +219,7 @@ func encodeFieldValue(fieldValue *modelv1.FieldValue) []byte {
return nil
}
-func decodeFieldValue(fieldValue []byte, fieldSpec *databasev1.FieldSpec) *modelv1.FieldValue {
+func DecodeFieldValue(fieldValue []byte, fieldSpec *databasev1.FieldSpec) *modelv1.FieldValue {
switch fieldSpec.GetFieldType() {
case databasev1.FieldType_FIELD_TYPE_STRING:
return &modelv1.FieldValue{Value: &modelv1.FieldValue_Str{Str: &modelv1.Str{Value: string(fieldValue)}}}
diff --git a/banyand/measure/measure_write_test.go b/banyand/measure/measure_write_test.go
index 935de8d..caf13ae 100644
--- a/banyand/measure/measure_write_test.go
+++ b/banyand/measure/measure_write_test.go
@@ -59,11 +59,15 @@ var _ = Describe("Write and Update service_cpm_minute", func() {
iters, err := seeker.Seek()
Expect(err).ShouldNot(HaveOccurred())
for _, iter := range iters {
- defer func(iterator tsdb.Iterator) {
+ defer func(iterator tsdb.ItemIterator) {
Expect(iterator.Close()).ShouldNot(HaveOccurred())
}(iter)
- for iter.Next() {
- num++
+ for {
+ if _, hasNext := iter.Next(); hasNext {
+ num++
+ } else {
+ break
+ }
}
}
}
diff --git a/banyand/measure/metadata.go b/banyand/measure/metadata.go
index 3f3c8da..fbbae42 100644
--- a/banyand/measure/metadata.go
+++ b/banyand/measure/metadata.go
@@ -14,6 +14,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package measure
import (
@@ -56,10 +57,10 @@ func newSchemaRepo(path string, metadata metadata.Repo, repo discovery.ServiceRe
}
}
-func (sr *schemaRepo) OnAddOrUpdate(m schema.Metadata) {
- switch m.Kind {
+func (sr *schemaRepo) OnAddOrUpdate(metadata schema.Metadata) {
+ switch metadata.Kind {
case schema.KindGroup:
- g := m.Spec.(*commonv1.Group)
+ g := metadata.Spec.(*commonv1.Group)
if g.Catalog != commonv1.Catalog_CATALOG_MEASURE {
return
}
@@ -72,10 +73,10 @@ func (sr *schemaRepo) OnAddOrUpdate(m schema.Metadata) {
sr.SendMetadataEvent(resourceSchema.MetadataEvent{
Typ: resourceSchema.EventAddOrUpdate,
Kind: resourceSchema.EventKindResource,
- Metadata: m.Spec.(*databasev1.Measure).GetMetadata(),
+ Metadata: metadata.Spec.(*databasev1.Measure).GetMetadata(),
})
case schema.KindIndexRuleBinding:
- irb, ok := m.Spec.(*databasev1.IndexRuleBinding)
+ irb, ok := metadata.Spec.(*databasev1.IndexRuleBinding)
if !ok {
sr.l.Warn().Msg("fail to convert message to IndexRuleBinding")
return
@@ -84,7 +85,7 @@ func (sr *schemaRepo) OnAddOrUpdate(m schema.Metadata) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
stm, err := sr.metadata.MeasureRegistry().GetMeasure(ctx, &commonv1.Metadata{
Name: irb.GetSubject().GetName(),
- Group: m.Group,
+ Group: metadata.Group,
})
cancel()
if err != nil {
@@ -99,8 +100,8 @@ func (sr *schemaRepo) OnAddOrUpdate(m schema.Metadata) {
}
case schema.KindIndexRule:
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
- subjects, err := sr.metadata.Subjects(ctx, m.Spec.(*databasev1.IndexRule), commonv1.Catalog_CATALOG_MEASURE)
- cancel()
+ defer cancel()
+ subjects, err := sr.metadata.Subjects(ctx, metadata.Spec.(*databasev1.IndexRule), commonv1.Catalog_CATALOG_MEASURE)
if err != nil {
sr.l.Error().Err(err).Msg("fail to get subjects(measure)")
return
@@ -112,14 +113,20 @@ func (sr *schemaRepo) OnAddOrUpdate(m schema.Metadata) {
Metadata: sub.(*databasev1.Measure).GetMetadata(),
})
}
+ case schema.KindTopNAggregation:
+ sr.SendMetadataEvent(resourceSchema.MetadataEvent{
+ Typ: resourceSchema.EventAddOrUpdate,
+ Kind: resourceSchema.EventKindResource,
+ Metadata: metadata.Spec.(*databasev1.TopNAggregation).GetSourceMeasure(),
+ })
default:
}
}
-func (sr *schemaRepo) OnDelete(m schema.Metadata) {
- switch m.Kind {
+func (sr *schemaRepo) OnDelete(metadata schema.Metadata) {
+ switch metadata.Kind {
case schema.KindGroup:
- g := m.Spec.(*commonv1.Group)
+ g := metadata.Spec.(*commonv1.Group)
if g.Catalog != commonv1.Catalog_CATALOG_MEASURE {
return
}
@@ -132,27 +139,35 @@ func (sr *schemaRepo) OnDelete(m schema.Metadata) {
sr.SendMetadataEvent(resourceSchema.MetadataEvent{
Typ: resourceSchema.EventDelete,
Kind: resourceSchema.EventKindResource,
- Metadata: m.Spec.(*databasev1.Measure).GetMetadata(),
+ Metadata: metadata.Spec.(*databasev1.Measure).GetMetadata(),
})
case schema.KindIndexRuleBinding:
- if m.Spec.(*databasev1.IndexRuleBinding).GetSubject().Catalog == commonv1.Catalog_CATALOG_MEASURE {
+ if metadata.Spec.(*databasev1.IndexRuleBinding).GetSubject().Catalog == commonv1.Catalog_CATALOG_MEASURE {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
- stm, err := sr.metadata.MeasureRegistry().GetMeasure(ctx, &commonv1.Metadata{
- Name: m.Name,
- Group: m.Group,
+ defer cancel()
+ m, err := sr.metadata.MeasureRegistry().GetMeasure(ctx, &commonv1.Metadata{
+ Name: metadata.Name,
+ Group: metadata.Group,
})
- cancel()
if err != nil {
sr.l.Error().Err(err).Msg("fail to get subject")
return
}
+ // we should update instead of delete
sr.SendMetadataEvent(resourceSchema.MetadataEvent{
- Typ: resourceSchema.EventDelete,
+ Typ: resourceSchema.EventAddOrUpdate,
Kind: resourceSchema.EventKindResource,
- Metadata: stm.GetMetadata(),
+ Metadata: m.GetMetadata(),
})
}
case schema.KindIndexRule:
+ case schema.KindTopNAggregation:
+ // we should update instead of delete
+ sr.SendMetadataEvent(resourceSchema.MetadataEvent{
+ Typ: resourceSchema.EventAddOrUpdate,
+ Kind: resourceSchema.EventKindResource,
+ Metadata: metadata.Spec.(*databasev1.TopNAggregation).GetSourceMeasure(),
+ })
default:
}
}
@@ -187,8 +202,9 @@ func newSupplier(path string, metadata metadata.Repo, dbOpts tsdb.DatabaseOpts,
func (s *supplier) OpenResource(shardNum uint32, db tsdb.Supplier, spec resourceSchema.ResourceSpec) (resourceSchema.Resource, error) {
measureSchema := spec.Schema.(*databasev1.Measure)
return openMeasure(shardNum, db, measureSpec{
- schema: measureSchema,
- indexRules: spec.IndexRules,
+ schema: measureSchema,
+ indexRules: spec.IndexRules,
+ topNAggregations: spec.Aggregations,
}, s.l)
}
diff --git a/banyand/measure/service.go b/banyand/measure/service.go
index 26219d2..191ebb6 100644
--- a/banyand/measure/service.go
+++ b/banyand/measure/service.go
@@ -31,6 +31,7 @@ import (
"github.com/apache/skywalking-banyandb/banyand/metadata/schema"
"github.com/apache/skywalking-banyandb/banyand/queue"
"github.com/apache/skywalking-banyandb/banyand/tsdb"
+ "github.com/apache/skywalking-banyandb/pkg/bus"
"github.com/apache/skywalking-banyandb/pkg/logger"
"github.com/apache/skywalking-banyandb/pkg/run"
resourceSchema "github.com/apache/skywalking-banyandb/pkg/schema"
@@ -54,12 +55,13 @@ type service struct {
root string
dbOpts tsdb.DatabaseOpts
- schemaRepo schemaRepo
- writeListener *writeCallback
- l *logger.Logger
- metadata metadata.Repo
- pipeline queue.Queue
- repo discovery.ServiceRepo
+ schemaRepo schemaRepo
+ writeListener bus.MessageListener
+ processListener bus.MessageListener
+ l *logger.Logger
+ metadata metadata.Repo
+ pipeline queue.Queue
+ repo discovery.ServiceRepo
// stop channel for the service
stopCh chan struct{}
}
@@ -108,29 +110,36 @@ func (s *service) PreRun() error {
if g.Catalog != commonv1.Catalog_CATALOG_MEASURE {
continue
}
- gp, err := s.schemaRepo.StoreGroup(g.Metadata)
- if err != nil {
- return err
+ gp, innerErr := s.schemaRepo.StoreGroup(g.Metadata)
+ if innerErr != nil {
+ return innerErr
}
ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
- schemas, err := s.metadata.MeasureRegistry().ListMeasure(ctx, schema.ListOpt{Group: gp.GetSchema().GetMetadata().Name})
+ allMeasureSchemas, innerErr := s.metadata.MeasureRegistry().
+ ListMeasure(ctx, schema.ListOpt{Group: gp.GetSchema().GetMetadata().GetName()})
cancel()
- if err != nil {
- return err
+ if innerErr != nil {
+ return innerErr
}
- for _, sa := range schemas {
- if _, innerErr := gp.StoreResource(sa); innerErr != nil {
+ for _, measureSchema := range allMeasureSchemas {
+ if _, innerErr := gp.StoreResource(measureSchema); innerErr != nil {
return innerErr
}
}
}
s.writeListener = setUpWriteCallback(s.l, &s.schemaRepo)
+ err = s.pipeline.Subscribe(data.TopicMeasureWrite, s.writeListener)
+ if err != nil {
+ return err
+ }
- errWrite := s.pipeline.Subscribe(data.TopicMeasureWrite, s.writeListener)
- if errWrite != nil {
- return errWrite
+ s.processListener = setUpStreamingProcessCallback(s.l, &s.schemaRepo)
+ err = s.pipeline.Subscribe(data.TopicMeasureWrite, s.processListener)
+ if err != nil {
+ return err
}
+
return nil
}
@@ -139,8 +148,9 @@ func (s *service) Serve() run.StopNotify {
// run a serial watcher
go s.schemaRepo.Watcher()
- s.metadata.MeasureRegistry().RegisterHandler(schema.KindGroup|schema.KindMeasure|schema.KindIndexRuleBinding|schema.KindIndexRule,
- &s.schemaRepo)
+ s.metadata.MeasureRegistry().
+ RegisterHandler(schema.KindGroup|schema.KindMeasure|schema.KindIndexRuleBinding|schema.KindIndexRule|schema.KindTopNAggregation,
+ &s.schemaRepo)
s.stopCh = make(chan struct{})
return s.stopCh
diff --git a/banyand/metadata/metadata.go b/banyand/metadata/metadata.go
index 8ecdb31..6ae2dd4 100644
--- a/banyand/metadata/metadata.go
+++ b/banyand/metadata/metadata.go
@@ -31,6 +31,12 @@ import (
"github.com/apache/skywalking-banyandb/pkg/run"
)
+// TopNFilter provides methods to find a specific topNAggregation which can be attached to the given resource.
+// Currently, the resource can only be Measure
+type TopNFilter interface {
+ TopNAggregations(ctx context.Context, source *commonv1.Metadata) ([]*databasev1.TopNAggregation, error)
+}
+
// IndexFilter provides methods to find a specific index related objects and vice versa
type IndexFilter interface {
// IndexRules fetches v1.IndexRule by subject defined in IndexRuleBinding
@@ -41,6 +47,7 @@ type IndexFilter interface {
type Repo interface {
IndexFilter
+ TopNFilter
StreamRegistry() schema.Stream
IndexRuleRegistry() schema.IndexRule
IndexRuleBindingRegistry() schema.IndexRuleBinding
@@ -224,6 +231,23 @@ func (s *service) Subjects(ctx context.Context, indexRule *databasev1.IndexRule,
return foundSubjects, subjectErr
}
+func (s *service) TopNAggregations(ctx context.Context, source *commonv1.Metadata) ([]*databasev1.TopNAggregation, error) {
+ aggregations, err := s.schemaRegistry.ListTopNAggregation(ctx, schema.ListOpt{Group: source.GetGroup()})
+ if err != nil {
+ return nil, err
+ }
+
+ var result []*databasev1.TopNAggregation
+ for _, aggrDef := range aggregations {
+ // filter sourceMeasure
+ if aggrDef.GetSourceMeasure().GetName() == source.GetName() {
+ result = append(result, aggrDef)
+ }
+ }
+
+ return result, nil
+}
+
func contains(s []string, e string) bool {
for _, a := range s {
if a == e {
diff --git a/banyand/metadata/schema/schema.go b/banyand/metadata/schema/schema.go
index 31898d5..7363b95 100644
--- a/banyand/metadata/schema/schema.go
+++ b/banyand/metadata/schema/schema.go
@@ -98,6 +98,8 @@ func (tm TypeMeta) Unmarshal(data []byte) (m proto.Message, err error) {
m = &databasev1.IndexRule{}
case KindProperty:
m = &propertyv1.Property{}
+ case KindTopNAggregation:
+ m = &databasev1.TopNAggregation{}
default:
return nil, ErrUnsupportedEntityType
}
@@ -134,6 +136,11 @@ func (m Metadata) Key() (string, error) {
Group: m.Group,
Name: m.Name,
}), nil
+ case KindTopNAggregation:
+ return formatTopNAggregationKey(&commonv1.Metadata{
+ Group: m.Group,
+ Name: m.Name,
+ }), nil
default:
return "", ErrUnsupportedEntityType
}
diff --git a/banyand/query/processor.go b/banyand/query/processor.go
index 1ad9026..aaaadc8 100644
--- a/banyand/query/processor.go
+++ b/banyand/query/processor.go
@@ -45,6 +45,7 @@ var (
_ Executor = (*queryService)(nil)
_ bus.MessageListener = (*streamQueryProcessor)(nil)
_ bus.MessageListener = (*measureQueryProcessor)(nil)
+ _ bus.MessageListener = (*topNQueryProcessor)(nil)
)
type queryService struct {
@@ -54,6 +55,7 @@ type queryService struct {
pipeline queue.Queue
sqp *streamQueryProcessor
mqp *measureQueryProcessor
+ tqp *topNQueryProcessor
}
type streamQueryProcessor struct {
@@ -118,7 +120,7 @@ type measureQueryProcessor struct {
func (p *measureQueryProcessor) Rev(message bus.Message) (resp bus.Message) {
queryCriteria, ok := message.Data().(*measurev1.QueryRequest)
if !ok {
- p.queryService.log.Warn().Msg("invalid event data type")
+ p.log.Warn().Msg("invalid event data type")
return
}
p.log.Debug().Msg("received a query event")
@@ -180,8 +182,9 @@ func (q *queryService) Name() string {
func (q *queryService) PreRun() error {
q.log = logger.GetLogger(moduleName)
- var err error
- err = multierr.Append(err, q.pipeline.Subscribe(data.TopicStreamQuery, q.sqp))
- err = multierr.Append(err, q.pipeline.Subscribe(data.TopicMeasureQuery, q.mqp))
- return err
+ return multierr.Combine(
+ q.pipeline.Subscribe(data.TopicStreamQuery, q.sqp),
+ q.pipeline.Subscribe(data.TopicMeasureQuery, q.mqp),
+ q.pipeline.Subscribe(data.TopicTopNQuery, q.tqp),
+ )
}
diff --git a/banyand/query/processor_test.go b/banyand/query/processor_test.go
index e8bc07e..7450bae 100644
--- a/banyand/query/processor_test.go
+++ b/banyand/query/processor_test.go
@@ -50,10 +50,8 @@ import (
)
var (
- svcs *services
- deferFn func()
- streamSchema stream.Stream
- sT, eT time.Time
+ svcs *services
+ deferFn func()
)
// BeforeSuite - Init logger
@@ -64,15 +62,6 @@ var _ = BeforeSuite(func() {
})).To(Succeed())
svcs, deferFn = setUpServices()
- var err error
- streamSchema, err = svcs.stream.Stream(&commonv1.Metadata{
- Name: "sw",
- Group: "default",
- })
- Expect(err).ShouldNot(HaveOccurred())
- baseTs := setUpStreamQueryData("multiple_shards.json", streamSchema)
-
- sT, eT = baseTs, baseTs.Add(1*time.Hour)
})
var _ = AfterSuite(func() {
@@ -180,7 +169,22 @@ func setUpStreamQueryData(dataFile string, stream stream.Stream) (baseTime time.
return baseTime
}
-var _ = Describe("Stream Query", func() {
+var _ = Describe("Stream Query", Ordered, func() {
+ var streamSchema stream.Stream
+ var sT, eT time.Time
+
+ BeforeAll(func() {
+ var err error
+ // setup stream data
+ streamSchema, err = svcs.stream.Stream(&commonv1.Metadata{
+ Name: "sw",
+ Group: "default",
+ })
+ Expect(err).ShouldNot(HaveOccurred())
+ baseTs := setUpStreamQueryData("multiple_shards.json", streamSchema)
+ sT, eT = baseTs, baseTs.Add(1*time.Hour)
+ })
+
It("should return nothing when querying given timeRange which is out of the time range of data", func() {
query := pb.NewStreamQueryRequestBuilder().
Limit(10).
diff --git a/banyand/query/processor_topn.go b/banyand/query/processor_topn.go
new file mode 100644
index 0000000..6461a43
--- /dev/null
+++ b/banyand/query/processor_topn.go
@@ -0,0 +1,298 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package query
+
+import (
+ "bytes"
+ "container/heap"
+ "context"
+ "math"
+ "time"
+
+ "github.com/pkg/errors"
+ "google.golang.org/protobuf/proto"
+
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/banyand/measure"
+ "github.com/apache/skywalking-banyandb/banyand/tsdb"
+ "github.com/apache/skywalking-banyandb/pkg/bus"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ "github.com/apache/skywalking-banyandb/pkg/flow/streaming"
+ "github.com/apache/skywalking-banyandb/pkg/query/aggregation"
+ "github.com/apache/skywalking-banyandb/pkg/timestamp"
+)
+
+type topNQueryProcessor struct {
+ measureService measure.Service
+ *queryService
+}
+
+func (t *topNQueryProcessor) Rev(message bus.Message) (resp bus.Message) {
+ request, ok := message.Data().(*measurev1.TopNRequest)
+ if !ok {
+ t.log.Warn().Msg("invalid event data type")
+ return
+ }
+ t.log.Info().Msg("received a topN query event")
+ topNMetadata := request.GetMetadata()
+ topNSchema, err := t.metaService.TopNAggregationRegistry().GetTopNAggregation(context.TODO(), topNMetadata)
+ if err != nil {
+ t.log.Error().Err(err).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to get execution context")
+ return
+ }
+ sourceMeasure, err := t.measureService.Measure(topNSchema.GetSourceMeasure())
+ if err != nil {
+ t.log.Error().Err(err).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to find source measure")
+ return
+ }
+ shards, err := sourceMeasure.CompanionShards(topNMetadata)
+ if err != nil {
+ t.log.Error().Err(err).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to list shards")
+ return
+ }
+ aggregator := createTopNPostAggregator(request.GetTopN(),
+ request.GetAgg(), request.GetFieldValueSort())
+ entity, err := locateEntity(topNSchema, request.GetConditions())
+ if err != nil {
+ t.log.Error().Err(err).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to parse entity")
+ return
+ }
+ for _, shard := range shards {
+ // TODO: support condition
+ sl, innerErr := shard.Series().List(tsdb.NewPath(entity))
+ if innerErr != nil {
+ t.log.Error().Err(innerErr).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to list series")
+ return
+ }
+ for _, series := range sl {
+ iters, scanErr := t.scanSeries(series, request)
+ if scanErr != nil {
+ t.log.Error().Err(innerErr).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to scan series")
+ return
+ }
+ for _, iter := range iters {
+ defer func(iter tsdb.ItemIterator) {
+ _ = iter.Close()
+ }(iter)
+ for {
+ if item, hasNext := iter.Next(); hasNext {
+ tuple, parseErr := parseTopNFamily(item, sourceMeasure.GetInterval())
+ if parseErr != nil {
+ t.log.Error().Err(parseErr).
+ Str("topN", topNMetadata.GetName()).
+ Msg("fail to parse topN family")
+ return
+ }
+ _ = aggregator.put(tuple.V1.(string), tuple.V2.(int64))
+ } else {
+ break
+ }
+ }
+ }
+ }
+ }
+
+ itemLen := int(math.Min(float64(request.GetTopN()), float64(aggregator.Len())))
+ topNItems := make([]*measurev1.TopNList_Item, 0, itemLen)
+
+ for _, item := range aggregator.items[0:itemLen] {
+ topNItems = append(topNItems, &measurev1.TopNList_Item{
+ Name: item.key,
+ Value: &modelv1.FieldValue{
+ Value: &modelv1.FieldValue_Int{
+ Int: &modelv1.Int{Value: item.int64Func.Val()},
+ },
+ },
+ })
+ }
+
+ now := time.Now().UnixNano()
+ resp = bus.NewMessage(bus.MessageID(now), topNItems)
+
+ return
+}
+
+func locateEntity(topNSchema *databasev1.TopNAggregation, conditions []*modelv1.Condition) (tsdb.Entity, error) {
+ entityMap := make(map[string]int)
+ entity := make([]tsdb.Entry, 1+len(topNSchema.GetGroupByTagNames()))
+ entity[0] = tsdb.AnyEntry
+ for idx, tagName := range topNSchema.GetGroupByTagNames() {
+ entityMap[tagName] = idx + 1
+ // fill AnyEntry by default
+ entity[idx+1] = tsdb.AnyEntry
+ }
+ for _, pairQuery := range conditions {
+ // TODO: check op?
+ if entityIdx, ok := entityMap[pairQuery.GetName()]; ok {
+ switch v := pairQuery.GetValue().GetValue().(type) {
+ case *modelv1.TagValue_Str:
+ entity[entityIdx] = []byte(v.Str.GetValue())
+ case *modelv1.TagValue_Id:
+ entity[entityIdx] = []byte(v.Id.GetValue())
+ case *modelv1.TagValue_Int:
+ entity[entityIdx] = convert.Int64ToBytes(v.Int.GetValue())
+ default:
+ return nil, errors.New("unsupported condition tag type for entity")
+ }
+ continue
+ }
+ return nil, errors.New("only groupBy tag name is supported")
+ }
+ return entity, nil
+}
+
+func parseTopNFamily(item tsdb.Item, interval time.Duration) (*streaming.Tuple2, error) {
+ familyRawBytes, err := item.Family(familyIdentity(measure.TopNTagFamily, measure.TagFlag))
+ if err != nil {
+ return nil, err
+ }
+ tagFamily := &modelv1.TagFamilyForWrite{}
+ err = proto.Unmarshal(familyRawBytes, tagFamily)
+ if err != nil {
+ return nil, err
+ }
+ fieldBytes, err := item.Family(familyIdentity(measure.TopNValueFieldSpec.GetName(),
+ measure.EncoderFieldFlag(measure.TopNValueFieldSpec, interval)))
+ if err != nil {
+ return nil, err
+ }
+ fieldValue := measure.DecodeFieldValue(fieldBytes, measure.TopNValueFieldSpec)
+ return &streaming.Tuple2{
+ // GroupValues
+ V1: tagFamily.GetTags()[1].GetStr().GetValue(),
+ // FieldValue
+ V2: fieldValue.GetInt().GetValue(),
+ }, nil
+}
+
+func familyIdentity(name string, flag []byte) []byte {
+ return bytes.Join([][]byte{tsdb.Hash([]byte(name)), flag}, nil)
+}
+
+func (t *topNQueryProcessor) scanSeries(series tsdb.Series, request *measurev1.TopNRequest) ([]tsdb.ItemIterator, error) {
+ seriesSpan, err := series.Span(timestamp.NewInclusiveTimeRange(
+ request.GetTimeRange().GetBegin().AsTime(),
+ request.GetTimeRange().GetEnd().AsTime()),
+ )
+ defer func(seriesSpan tsdb.SeriesSpan) {
+ _ = seriesSpan.Close()
+ }(seriesSpan)
+ if err != nil {
+ return nil, err
+ }
+ seeker, err := seriesSpan.SeekerBuilder().OrderByTime(modelv1.Sort_SORT_ASC).Build()
+ if err != nil {
+ return nil, err
+ }
+ return seeker.Seek()
+}
+
+var _ heap.Interface = (*topNPostAggregator)(nil)
+
+type topNAggregatorItem struct {
+ key string
+ int64Func aggregation.Int64Func
+ index int
+}
+
+type topNPostAggregator struct {
+ topN int32
+ sort modelv1.Sort
+ aggrFunc modelv1.AggregationFunction
+ items []*topNAggregatorItem
+ cache map[string]*topNAggregatorItem
+}
+
+func (aggr topNPostAggregator) Len() int {
+ return len(aggr.items)
+}
+
+func (aggr topNPostAggregator) Less(i, j int) bool {
+ if aggr.sort == modelv1.Sort_SORT_DESC {
+ return aggr.items[i].int64Func.Val() > aggr.items[j].int64Func.Val()
+ }
+ return aggr.items[i].int64Func.Val() < aggr.items[j].int64Func.Val()
+}
+
+func (aggr *topNPostAggregator) Swap(i, j int) {
+ aggr.items[i], aggr.items[j] = aggr.items[j], aggr.items[i]
+ aggr.items[i].index = i
+ aggr.items[j].index = j
+}
+
+func (aggr *topNPostAggregator) Push(x any) {
+ n := len(aggr.items)
+ item := x.(*topNAggregatorItem)
+ item.index = n
+ aggr.items = append(aggr.items, item)
+}
+
+func (aggr *topNPostAggregator) Pop() any {
+ old := aggr.items
+ n := len(old)
+ item := old[n-1]
+ old[n-1] = nil
+ item.index = -1
+ aggr.items = old[0 : n-1]
+ return item
+}
+
+func createTopNPostAggregator(topN int32, aggrFunc modelv1.AggregationFunction, sort modelv1.Sort) *topNPostAggregator {
+ aggregator := &topNPostAggregator{
+ topN: topN,
+ sort: sort,
+ aggrFunc: aggrFunc,
+ cache: make(map[string]*topNAggregatorItem),
+ }
+ heap.Init(aggregator)
+ return aggregator
+}
+
+func (aggr *topNPostAggregator) put(key string, val int64) error {
+ if item, found := aggr.cache[key]; found {
+ item.int64Func.In(val)
+ heap.Fix(aggr, item.index)
+ return nil
+ }
+ aggrFunc, err := aggregation.NewInt64Func(aggr.aggrFunc)
+ if err != nil {
+ return err
+ }
+ item := &topNAggregatorItem{
+ key: key,
+ int64Func: aggrFunc,
+ }
+ item.int64Func.In(val)
+ aggr.cache[key] = item
+ heap.Push(aggr, item)
+ return nil
+}
diff --git a/banyand/query/processor_topn_test.go b/banyand/query/processor_topn_test.go
new file mode 100644
index 0000000..6c3cb64
--- /dev/null
+++ b/banyand/query/processor_topn_test.go
@@ -0,0 +1,165 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package query_test
+
+import (
+ "encoding/json"
+ "time"
+
+ . "github.com/onsi/ginkgo/v2"
+ . "github.com/onsi/gomega"
+ "google.golang.org/protobuf/encoding/protojson"
+ "google.golang.org/protobuf/types/known/timestamppb"
+
+ "github.com/apache/skywalking-banyandb/api/data"
+ commonv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/common/v1"
+ measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
+ "github.com/apache/skywalking-banyandb/banyand/measure"
+ "github.com/apache/skywalking-banyandb/pkg/bus"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/timestamp"
+)
+
+func setupMeasureQueryData(dataFile string, measureSchema measure.Measure,
+ interval time.Duration, lastDelay time.Duration,
+) (baseTime time.Time) {
+ var templates []interface{}
+ baseTime = timestamp.NowMilli()
+ content, err := dataFS.ReadFile("testdata/" + dataFile)
+ Expect(err).ShouldNot(HaveOccurred())
+ Expect(json.Unmarshal(content, &templates)).Should(Succeed())
+ for i, template := range templates {
+ rawDataPointValue, errMarshal := json.Marshal(template)
+ Expect(errMarshal).ShouldNot(HaveOccurred())
+ dataPointValue := &measurev1.DataPointValue{}
+ Expect(protojson.Unmarshal(rawDataPointValue, dataPointValue)).ShouldNot(HaveOccurred())
+ dataPointValue.Timestamp = timestamppb.New(baseTime.Add(time.Duration(i) * interval))
+ errInner := measureSchema.Write(dataPointValue)
+ Expect(errInner).ShouldNot(HaveOccurred())
+ if i == len(templates)-1 {
+ // add delay
+ dataPointValue.Timestamp.Seconds += int64(lastDelay.Seconds())
+ lastErr := measureSchema.Write(dataPointValue)
+ Expect(lastErr).ShouldNot(HaveOccurred())
+ }
+ }
+ return baseTime
+}
+
+var _ = Describe("TopN Query", Ordered, func() {
+ var measureSchema measure.Measure
+ var sT, eT time.Time
+
+ BeforeAll(func() {
+ var err error
+ // setup measure data
+ measureSchema, err = svcs.measure.Measure(&commonv1.Metadata{
+ Name: "service_cpm_minute",
+ Group: "sw_metric",
+ })
+ Expect(err).ShouldNot(HaveOccurred())
+ baseTs := setupMeasureQueryData("service_cpm_minute_data.json", measureSchema, 15*time.Second,
+ 2*time.Minute)
+ sT, eT = baseTs.Add(-1*time.Minute), baseTs.Add(10*time.Minute)
+ })
+
+ It("Query without condition", func() {
+ query := pbv1.NewMeasureTopNRequestBuilder().
+ Metadata("sw_metric", "service_cpm_minute_top100").
+ TimeRange(sT, eT).
+ TopN(1).
+ Max().
+ Build()
+ Eventually(func(g Gomega) int64 {
+ now := time.Now()
+ msg := bus.NewMessage(bus.MessageID(now.UnixNano()), query)
+ f, err := svcs.pipeline.Publish(data.TopicTopNQuery, msg)
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(f).ShouldNot(BeNil())
+ resp, err := f.Get()
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(resp).ShouldNot(BeNil())
+ g.Expect(resp.Data()).Should(HaveLen(1))
+ return resp.Data().([]*measurev1.TopNList_Item)[0].GetValue().GetInt().GetValue()
+ }).WithTimeout(time.Second * 10).Should(BeNumerically("==", 5))
+ })
+
+ It("Query with condition and min aggregation", func() {
+ query := pbv1.NewMeasureTopNRequestBuilder().
+ Metadata("sw_metric", "service_cpm_minute_top100").
+ TimeRange(sT, eT).
+ TopN(1).
+ Min().
+ Build()
+ Eventually(func(g Gomega) int64 {
+ now := time.Now()
+ msg := bus.NewMessage(bus.MessageID(now.UnixNano()), query)
+ f, err := svcs.pipeline.Publish(data.TopicTopNQuery, msg)
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(f).ShouldNot(BeNil())
+ resp, err := f.Get()
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(resp).ShouldNot(BeNil())
+ g.Expect(resp.Data()).Should(HaveLen(1))
+ return resp.Data().([]*measurev1.TopNList_Item)[0].GetValue().GetInt().GetValue()
+ }).WithTimeout(time.Second * 10).Should(BeNumerically("==", 1))
+ })
+
+ It("Query with condition", func() {
+ query := pbv1.NewMeasureTopNRequestBuilder().
+ Metadata("sw_metric", "service_cpm_minute_top100").
+ TimeRange(sT, eT).
+ Conditions(pbv1.Eq("entity_id", "entity_2")).
+ TopN(1).
+ Max().
+ Build()
+ Eventually(func(g Gomega) int64 {
+ now := time.Now()
+ msg := bus.NewMessage(bus.MessageID(now.UnixNano()), query)
+ f, err := svcs.pipeline.Publish(data.TopicTopNQuery, msg)
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(f).ShouldNot(BeNil())
+ resp, err := f.Get()
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(resp).ShouldNot(BeNil())
+ g.Expect(resp.Data()).Should(HaveLen(1))
+ return resp.Data().([]*measurev1.TopNList_Item)[0].GetValue().GetInt().GetValue()
+ }).WithTimeout(time.Second * 10).Should(BeNumerically("==", 5))
+ })
+
+ It("Query with condition for non-group-by topN", func() {
+ query := pbv1.NewMeasureTopNRequestBuilder().
+ Metadata("sw_metric", "service_cpm_minute_no_group_by_top100").
+ TimeRange(sT, eT).
+ TopN(1).
+ Max().
+ Build()
+ Eventually(func(g Gomega) int64 {
+ now := time.Now()
+ msg := bus.NewMessage(bus.MessageID(now.UnixNano()), query)
+ f, err := svcs.pipeline.Publish(data.TopicTopNQuery, msg)
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(f).ShouldNot(BeNil())
+ resp, err := f.Get()
+ g.Expect(err).ShouldNot(HaveOccurred())
+ g.Expect(resp).ShouldNot(BeNil())
+ g.Expect(resp.Data()).Should(HaveLen(1))
+ return resp.Data().([]*measurev1.TopNList_Item)[0].GetValue().GetInt().GetValue()
+ }).WithTimeout(time.Second * 10).Should(BeNumerically("==", 5))
+ })
+})
diff --git a/banyand/query/query.go b/banyand/query/query.go
index 1ef4d5f..267a922 100644
--- a/banyand/query/query.go
+++ b/banyand/query/query.go
@@ -40,13 +40,20 @@ func NewExecutor(_ context.Context, streamService stream.Service, measureService
serviceRepo: serviceRepo,
pipeline: pipeline,
}
+ // measure query processor
svc.mqp = &measureQueryProcessor{
measureService: measureService,
queryService: svc,
}
+ // stream query processor
svc.sqp = &streamQueryProcessor{
streamService: streamService,
queryService: svc,
}
+ // topN query processor
+ svc.tqp = &topNQueryProcessor{
+ measureService: measureService,
+ queryService: svc,
+ }
return svc, nil
}
diff --git a/banyand/query/testdata/service_cpm_minute_data.json b/banyand/query/testdata/service_cpm_minute_data.json
new file mode 100644
index 0000000..d29da17
--- /dev/null
+++ b/banyand/query/testdata/service_cpm_minute_data.json
@@ -0,0 +1,182 @@
+[
+ {
+ "tag_families": [
+ {
+ "tags": [
+ {
+ "id": {
+ "value": "1"
+ }
+ },
+ {
+ "str": {
+ "value": "entity_1"
+ }
+ }
+ ]
+ }
+ ],
+ "fields": [
+ {
+ "int": {
+ "value": 100
+ }
+ },
+ {
+ "int": {
+ "value": 1
+ }
+ }
+ ]
+ },
+ {
+ "tag_families": [
+ {
+ "tags": [
+ {
+ "id": {
+ "value": "2"
+ }
+ },
+ {
+ "str": {
+ "value": "entity_2"
+ }
+ }
+ ]
+ }
+ ],
+ "fields": [
+ {
+ "int": {
+ "value": 300
+ }
+ },
+ {
+ "int": {
+ "value": 5
+ }
+ }
+ ]
+ },
+ {
+ "tag_families": [
+ {
+ "tags": [
+ {
+ "id": {
+ "value": "1"
+ }
+ },
+ {
+ "str": {
+ "value": "entity_1"
+ }
+ }
+ ]
+ }
+ ],
+ "fields": [
+ {
+ "int": {
+ "value": 50
+ }
+ },
+ {
+ "int": {
+ "value": 4
+ }
+ }
+ ]
+ },
+ {
+ "tag_families": [
+ {
+ "tags": [
+ {
+ "id": {
+ "value": "1"
+ }
+ },
+ {
+ "str": {
+ "value": "entity_1"
+ }
+ }
+ ]
+ }
+ ],
+ "fields": [
+ {
+ "int": {
+ "value": 300
+ }
+ },
+ {
+ "int": {
+ "value": 5
+ }
+ }
+ ]
+ },
+ {
+ "tag_families": [
+ {
+ "tags": [
+ {
+ "id": {
+ "value": "2"
+ }
+ },
+ {
+ "str": {
+ "value": "entity_2"
+ }
+ }
+ ]
+ }
+ ],
+ "fields": [
+ {
+ "int": {
+ "value": 300
+ }
+ },
+ {
+ "int": {
+ "value": 5
+ }
+ }
+ ]
+ },
+ {
+ "tag_families": [
+ {
+ "tags": [
+ {
+ "id": {
+ "value": "2"
+ }
+ },
+ {
+ "str": {
+ "value": "entity_2"
+ }
+ }
+ ]
+ }
+ ],
+ "fields": [
+ {
+ "int": {
+ "value": 300
+ }
+ },
+ {
+ "int": {
+ "value": 5
+ }
+ }
+ ]
+ }
+]
\ No newline at end of file
diff --git a/banyand/stream/stream_query_test.go b/banyand/stream/stream_query_test.go
index 9735e4f..d9b7c2a 100644
--- a/banyand/stream/stream_query_test.go
+++ b/banyand/stream/stream_query_test.go
@@ -939,8 +939,12 @@ func queryData(s *stream, baseTime time.Time, opts queryOpts) (shardsForTest, er
}
for dataFlowID, iterator := range iter {
var elements []string
- for iterator.Next() {
- tagFamily, errInner := s.ParseTagFamily("searchable", iterator.Val())
+ for {
+ val, hasNext := iterator.Next()
+ if !hasNext {
+ break
+ }
+ tagFamily, errInner := s.ParseTagFamily("searchable", val)
if errInner != nil {
return nil, errInner
}
@@ -949,7 +953,7 @@ func queryData(s *stream, baseTime time.Time, opts queryOpts) (shardsForTest, er
elements = append(elements, tag.GetValue().GetStr().GetValue())
}
}
- eleID, errInner := s.ParseElementID(iterator.Val())
+ eleID, errInner := s.ParseElementID(val)
if errInner != nil {
return nil, errInner
}
diff --git a/banyand/stream/stream_write_test.go b/banyand/stream/stream_write_test.go
index c4abea4..c331fd0 100644
--- a/banyand/stream/stream_write_test.go
+++ b/banyand/stream/stream_write_test.go
@@ -227,8 +227,12 @@ var _ = Describe("Write", func() {
iters, errInternal := sb.Seek()
Expect(errInternal).ShouldNot(HaveOccurred())
for _, iter := range iters {
- for iter.Next() {
- eleSize++
+ for {
+ if _, hasNext := iter.Next(); hasNext {
+ eleSize++
+ } else {
+ break
+ }
}
iter.Close()
}
diff --git a/banyand/tsdb/block.go b/banyand/tsdb/block.go
index 1dc1df0..ccae9db 100644
--- a/banyand/tsdb/block.go
+++ b/banyand/tsdb/block.go
@@ -180,7 +180,7 @@ func (b *block) open() (err error) {
return nil
}
-func (b *block) delegate() (blockDelegate, error) {
+func (b *block) delegate() (BlockDelegate, error) {
if b.incRef() {
return &bDelegate{
delegate: b,
@@ -290,7 +290,7 @@ func (b *block) stats() (names []string, stats []observability.Statistics) {
return names, stats
}
-type blockDelegate interface {
+type BlockDelegate interface {
io.Closer
contains(ts time.Time) bool
write(key []byte, val []byte, ts time.Time) error
@@ -306,7 +306,7 @@ type blockDelegate interface {
String() string
}
-var _ blockDelegate = (*bDelegate)(nil)
+var _ BlockDelegate = (*bDelegate)(nil)
type bDelegate struct {
delegate *block
diff --git a/banyand/tsdb/scope.go b/banyand/tsdb/scope.go
index 8233d9a..134065d 100644
--- a/banyand/tsdb/scope.go
+++ b/banyand/tsdb/scope.go
@@ -87,5 +87,5 @@ func (sdd *scopedSeriesDatabase) Get(entity Entity) (Series, error) {
}
func (sdd *scopedSeriesDatabase) List(path Path) (SeriesList, error) {
- return sdd.delegated.List(path.Prepand(sdd.scope))
+ return sdd.delegated.List(path.Prepend(sdd.scope))
}
diff --git a/banyand/tsdb/segment.go b/banyand/tsdb/segment.go
index be2cb7a..d33bd33 100644
--- a/banyand/tsdb/segment.go
+++ b/banyand/tsdb/segment.go
@@ -225,14 +225,14 @@ func (bc *blockController) Parse(value string) (time.Time, error) {
panic("invalid interval unit")
}
-func (bc *blockController) span(timeRange timestamp.TimeRange) ([]blockDelegate, error) {
+func (bc *blockController) span(timeRange timestamp.TimeRange) ([]BlockDelegate, error) {
bb := bc.search(func(b *block) bool {
return b.Overlapping(timeRange)
})
if bb == nil {
return nil, nil
}
- dd := make([]blockDelegate, len(bb))
+ dd := make([]BlockDelegate, len(bb))
for i, b := range bb {
d, err := b.delegate()
if err != nil {
@@ -243,7 +243,7 @@ func (bc *blockController) span(timeRange timestamp.TimeRange) ([]blockDelegate,
return dd, nil
}
-func (bc *blockController) get(blockID uint16) (blockDelegate, error) {
+func (bc *blockController) get(blockID uint16) (BlockDelegate, error) {
b := bc.getBlock(blockID)
if b != nil {
return b.delegate()
diff --git a/banyand/tsdb/series.go b/banyand/tsdb/series.go
index 1696616..b44d1dc 100644
--- a/banyand/tsdb/series.go
+++ b/banyand/tsdb/series.go
@@ -27,7 +27,9 @@ import (
"go.uber.org/multierr"
"github.com/apache/skywalking-banyandb/api/common"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
"github.com/apache/skywalking-banyandb/pkg/convert"
+ "github.com/apache/skywalking-banyandb/pkg/index"
"github.com/apache/skywalking-banyandb/pkg/logger"
"github.com/apache/skywalking-banyandb/pkg/timestamp"
)
@@ -83,6 +85,10 @@ type SeriesSpan interface {
io.Closer
WriterBuilder() WriterBuilder
SeekerBuilder() SeekerBuilder
+ Blocks() []BlockDelegate
+ SeriesID() common.SeriesID
+ BuildItemIterators(BlockDelegate, ItemFilter) (ItemIterator, error)
+ BuildConditionalFilter(BlockDelegate, LogicalIndexedCondition) (ItemFilter, error)
}
var _ Series = (*series)(nil)
@@ -145,13 +151,46 @@ func newSeries(ctx context.Context, id common.SeriesID, blockDB blockDatabase) *
var _ SeriesSpan = (*seriesSpan)(nil)
type seriesSpan struct {
- blocks []blockDelegate
+ blocks []BlockDelegate
seriesID common.SeriesID
shardID common.ShardID
timeRange timestamp.TimeRange
l *logger.Logger
}
+func (s *seriesSpan) BuildItemIterators(b BlockDelegate, filter ItemFilter) (ItemIterator, error) {
+ termRange := index.RangeOpts{
+ Lower: convert.Int64ToBytes(s.timeRange.Start.UnixNano()),
+ Upper: convert.Int64ToBytes(s.timeRange.End.UnixNano()),
+ IncludesLower: true,
+ }
+ sortFieldIter, err := b.primaryIndexReader().
+ Iterator(
+ index.FieldKey{
+ SeriesID: s.seriesID,
+ },
+ termRange,
+ // TODO: support sort
+ modelv1.Sort_SORT_ASC,
+ )
+ if err != nil {
+ return nil, err
+ }
+ return newSearcherIterator(s.l, sortFieldIter, b.dataReader(), s.seriesID, []FilterFn{filter.Predicate}), nil
+}
+
+func (s *seriesSpan) BuildConditionalFilter(b BlockDelegate, lc LogicalIndexedCondition) (ItemFilter, error) {
+ return lc.build(s, b)
+}
+
+func (s *seriesSpan) SeriesID() common.SeriesID {
+ return s.seriesID
+}
+
+func (s *seriesSpan) Blocks() []BlockDelegate {
+ return s.blocks
+}
+
func (s *seriesSpan) Close() (err error) {
for _, delegate := range s.blocks {
err = multierr.Append(err, delegate.Close())
@@ -167,7 +206,7 @@ func (s *seriesSpan) SeekerBuilder() SeekerBuilder {
return newSeekerBuilder(s)
}
-func newSeriesSpan(ctx context.Context, timeRange timestamp.TimeRange, blocks []blockDelegate, id common.SeriesID, shardID common.ShardID) *seriesSpan {
+func newSeriesSpan(ctx context.Context, timeRange timestamp.TimeRange, blocks []BlockDelegate, id common.SeriesID, shardID common.ShardID) *seriesSpan {
s := &seriesSpan{
blocks: blocks,
seriesID: id,
diff --git a/banyand/tsdb/series_filter.go b/banyand/tsdb/series_filter.go
new file mode 100644
index 0000000..ecc368a
--- /dev/null
+++ b/banyand/tsdb/series_filter.go
@@ -0,0 +1,278 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package tsdb
+
+import (
+ "github.com/apache/skywalking-banyandb/api/common"
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ "github.com/apache/skywalking-banyandb/pkg/index"
+ "github.com/apache/skywalking-banyandb/pkg/index/posting"
+)
+
+var (
+ _ LogicalIndexedCondition = (*indexedCondition)(nil)
+ _ LogicalIndexedCondition = (*andIndexedConditions)(nil)
+ _ LogicalIndexedCondition = (*orIndexedConditions)(nil)
+)
+
+type LogicalIndexedCondition interface {
+ build(seriesSpan SeriesSpan, b BlockDelegate) (ItemFilter, error)
+}
+
+type indexedCondition struct {
+ indexRuleType databasev1.IndexRule_Type
+ indexRuleID uint32
+ condition Condition
+}
+
+func NewConditionWithIndexRule(condition Condition, indexRule *databasev1.IndexRule) LogicalIndexedCondition {
+ return &indexedCondition{
+ condition: condition,
+ indexRuleID: indexRule.GetMetadata().GetId(),
+ indexRuleType: indexRule.GetType(),
+ }
+}
+
+func (i *indexedCondition) buildConditions(seriesID common.SeriesID) index.Condition {
+ cond := make(index.Condition)
+ term := index.FieldKey{
+ IndexRuleID: i.indexRuleID,
+ SeriesID: seriesID,
+ }
+ for _, cv := range i.condition {
+ cond[term] = cv
+ break
+ }
+ return cond
+}
+
+func (i *indexedCondition) build(seriesSpan SeriesSpan, b BlockDelegate) (ItemFilter, error) {
+ seriesID := seriesSpan.SeriesID()
+ switch i.indexRuleType {
+ case databasev1.IndexRule_TYPE_INVERTED:
+ list, valid, innerErr := buildSingleFilterSet(b.invertedIndexReader(), i.buildConditions(seriesID), seriesID, i.indexRuleID)
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ return &roaringBitmapFilter{
+ list: list,
+ valid: valid,
+ }, nil
+ case databasev1.IndexRule_TYPE_TREE:
+ list, valid, innerErr := buildSingleFilterSet(b.lsmIndexReader(), i.buildConditions(seriesID), seriesID, i.indexRuleID)
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ return &roaringBitmapFilter{
+ list: list,
+ valid: valid,
+ }, nil
+ default:
+ return nil, ErrUnsupportedIndexRule
+ }
+}
+
+type andIndexedConditions []LogicalIndexedCondition
+
+func AndConditions(conditions ...LogicalIndexedCondition) LogicalIndexedCondition {
+ return andIndexedConditions(conditions)
+}
+
+func (a andIndexedConditions) build(seriesSpan SeriesSpan, b BlockDelegate) (ItemFilter, error) {
+ var singleFilter ItemFilter
+ for _, child := range a {
+ filter, err := child.build(seriesSpan, b)
+ if err != nil {
+ return nil, err
+ }
+ if singleFilter == nil {
+ singleFilter = filter
+ } else {
+ var innerErr error
+ singleFilter, innerErr = singleFilter.And(filter)
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ }
+ }
+ if singleFilter == nil {
+ return TrueFilter(), nil
+ }
+ return singleFilter, nil
+}
+
+type orIndexedConditions []LogicalIndexedCondition
+
+func OrConditions(conditions ...LogicalIndexedCondition) LogicalIndexedCondition {
+ return orIndexedConditions(conditions)
+}
+
+func (o orIndexedConditions) build(seriesSpan SeriesSpan, b BlockDelegate) (ItemFilter, error) {
+ var singleFilter ItemFilter
+ for _, child := range o {
+ filter, err := child.build(seriesSpan, b)
+ if err != nil {
+ return nil, err
+ }
+ if singleFilter == nil {
+ singleFilter = filter
+ } else {
+ var innerErr error
+ singleFilter, innerErr = singleFilter.Or(filter)
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ }
+ }
+ if singleFilter == nil {
+ return TrueFilter(), nil
+ }
+ return singleFilter, nil
+}
+
+func TrueFilter() ItemFilter {
+ return FilterFn(func(item Item) bool {
+ return true
+ })
+}
+
+func FalseFilter() ItemFilter {
+ return FilterFn(func(item Item) bool {
+ return false
+ })
+}
+
+func EmptyFilter() ItemFilter {
+ return FilterFn(nil)
+}
+
+type ItemFilter interface {
+ Predicate(Item) bool
+ And(filters ...ItemFilter) (ItemFilter, error)
+ Or(filters ...ItemFilter) (ItemFilter, error)
+}
+
+type roaringBitmapFilter struct {
+ list posting.List
+ valid bool
+}
+
+func (r *roaringBitmapFilter) Predicate(i Item) bool {
+ return r.list.Contains(i.ID())
+}
+
+func (r *roaringBitmapFilter) And(filters ...ItemFilter) (ItemFilter, error) {
+ allList := r.list
+ for _, filter := range filters {
+ if rbf, ok := filter.(*roaringBitmapFilter); ok {
+ innerErr := allList.Intersect(rbf.list)
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ // if one of the list is empty, it would always return false
+ if allList.IsEmpty() {
+ return FilterFn(func(_ Item) bool {
+ return false
+ }), nil
+ }
+ }
+ }
+ return &roaringBitmapFilter{
+ list: allList,
+ }, nil
+}
+
+func (r *roaringBitmapFilter) Or(filters ...ItemFilter) (ItemFilter, error) {
+ allList := r.list
+ for _, filter := range filters {
+ if rbf, ok := filter.(*roaringBitmapFilter); ok {
+ innerErr := allList.Union(rbf.list)
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ }
+ }
+ return &roaringBitmapFilter{
+ list: allList,
+ }, nil
+}
+
+func (f FilterFn) And(filters ...ItemFilter) (ItemFilter, error) {
+ if f == nil {
+ return and(filters), nil
+ }
+ return and(append([]ItemFilter{f}, filters...)), nil
+}
+
+func (f FilterFn) Or(filters ...ItemFilter) (ItemFilter, error) {
+ if f == nil {
+ return or(filters), nil
+ }
+ return or(append([]ItemFilter{f}, filters...)), nil
+}
+
+func (f FilterFn) Predicate(item Item) bool {
+ if f == nil {
+ panic("Call Predicate on a nil object will cause undefined behavior")
+ }
+ return f(item)
+}
+
+type and []ItemFilter
+
+func (allFilters and) Predicate(item Item) bool {
+ for _, filter := range allFilters {
+ if !filter.Predicate(item) {
+ return false
+ }
+ }
+ return true
+}
+
+func (allFilters and) And(filters ...ItemFilter) (ItemFilter, error) {
+ return append(allFilters, filters...), nil
+}
+
+func (allFilters and) Or(filters ...ItemFilter) (ItemFilter, error) {
+ if len(allFilters) == 0 {
+ return or(filters), nil
+ }
+ return or(append([]ItemFilter{allFilters}, filters...)), nil
+}
+
+type or []ItemFilter
+
+func (allFilters or) Predicate(item Item) bool {
+ for _, filter := range allFilters {
+ if filter.Predicate(item) {
+ return true
+ }
+ }
+ return false
+}
+
+func (allFilters or) And(filters ...ItemFilter) (ItemFilter, error) {
+ if len(filters) == 0 {
+ return and(filters), nil
+ }
+ return and(append([]ItemFilter{allFilters}, filters...)), nil
+}
+
+func (allFilters or) Or(filters ...ItemFilter) (ItemFilter, error) {
+ return append(allFilters, filters...), nil
+}
diff --git a/banyand/tsdb/series_filter_test.go b/banyand/tsdb/series_filter_test.go
new file mode 100644
index 0000000..24bee74
--- /dev/null
+++ b/banyand/tsdb/series_filter_test.go
@@ -0,0 +1,315 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package tsdb
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/pkg/index/posting"
+ "github.com/apache/skywalking-banyandb/pkg/index/posting/roaring"
+)
+
+func newItem(itemID common.ItemID) *item {
+ return &item{
+ itemID: itemID,
+ }
+}
+
+func Test_Filter_LogicalOperations(t *testing.T) {
+ tests := []struct {
+ name string
+ filterFactory func() (ItemFilter, error)
+ positiveItems []Item
+ negativeItems []Item
+ }{
+ {
+ name: "Empty AND FilterFn",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := EmptyFilter().And(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(0)},
+ negativeItems: []Item{newItem(1)},
+ },
+ {
+ name: "Empty OR FilterFn",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := EmptyFilter().Or(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(0)},
+ negativeItems: []Item{newItem(1)},
+ },
+ {
+ name: "(Empty AND FilterFn) OR FilterFn",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := EmptyFilter().And(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ filterFunc, err = filterFunc.Or(FilterFn(func(item Item) bool {
+ return item.ID()%3 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(0), newItem(2), newItem(3)},
+ negativeItems: []Item{newItem(5)},
+ },
+ {
+ name: "(Empty OR FilterFn) AND FilterFn",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := EmptyFilter().Or(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ filterFunc, err = filterFunc.And(FilterFn(func(item Item) bool {
+ return item.ID()%3 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(6), newItem(12), newItem(18)},
+ negativeItems: []Item{newItem(2), newItem(3)},
+ },
+ {
+ name: "(Empty AND FilterFn...) OR FilterFn...",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := EmptyFilter().And(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }), FilterFn(func(item Item) bool {
+ return item.ID()%3 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ filterFunc, err = filterFunc.Or(FilterFn(func(item Item) bool {
+ return item.ID()%5 == 0
+ }), FilterFn(func(item Item) bool {
+ return item.ID()%7 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(5), newItem(6), newItem(7), newItem(12)},
+ negativeItems: []Item{newItem(2), newItem(13)},
+ },
+ {
+ name: "(Empty OR FilterFn...) AND FilterFn...",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := EmptyFilter().Or(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }), FilterFn(func(item Item) bool {
+ return item.ID()%3 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ filterFunc, err = filterFunc.And(FilterFn(func(item Item) bool {
+ return item.ID()%5 == 0
+ }), FilterFn(func(item Item) bool {
+ return item.ID()%7 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(70), newItem(105)},
+ negativeItems: []Item{newItem(2), newItem(3), newItem(35)},
+ },
+ {
+ name: "Empty(and{}) AND FilterFn",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := ItemFilter(and{}).And(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(0)},
+ negativeItems: []Item{newItem(1)},
+ },
+ {
+ name: "Empty(or{}) OR FilterFn",
+ filterFactory: func() (ItemFilter, error) {
+ filterFunc, err := ItemFilter(or{}).Or(FilterFn(func(item Item) bool {
+ return item.ID()%2 == 0
+ }))
+ if err != nil {
+ return nil, err
+ }
+ return filterFunc, nil
+ },
+ positiveItems: []Item{newItem(0)},
+ negativeItems: []Item{newItem(1)},
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ assert := assert.New(t)
+ filterFunc, err := tt.filterFactory()
+ assert.NoError(err)
+ assert.NotNil(filterFunc)
+ for _, pi := range tt.positiveItems {
+ assert.True(filterFunc.Predicate(pi))
+ }
+ for _, ni := range tt.negativeItems {
+ assert.False(filterFunc.Predicate(ni))
+ }
+ })
+ }
+}
+
+var (
+ _ ItemFilter = (*mockConditionFactory)(nil)
+ _ LogicalIndexedCondition = (*mockConditionFactory)(nil)
+)
+
+type mockConditionFactory struct {
+ list posting.List
+}
+
+func (m *mockConditionFactory) build(seriesSpan SeriesSpan, b BlockDelegate) (ItemFilter, error) {
+ return &roaringBitmapFilter{list: m.list}, nil
+}
+
+func (m *mockConditionFactory) Predicate(i Item) bool {
+ return m.list.Contains(i.ID())
+}
+
+func (m *mockConditionFactory) And(filters ...ItemFilter) (ItemFilter, error) {
+ allList := m.list
+ for _, filter := range filters {
+ if rbf, ok := filter.(*roaringBitmapFilter); ok {
+ innerErr := allList.Intersect(rbf.list)
+ return nil, innerErr
+ }
+ }
+ return &mockConditionFactory{
+ list: allList,
+ }, nil
+}
+
+func (m *mockConditionFactory) Or(filters ...ItemFilter) (ItemFilter, error) {
+ allList := m.list
+ for _, filter := range filters {
+ if rbf, ok := filter.(*roaringBitmapFilter); ok {
+ innerErr := allList.Union(rbf.list)
+ return nil, innerErr
+ }
+ }
+ return &mockConditionFactory{
+ list: allList,
+ }, nil
+}
+
+func Test_RoaringBitmapFilter(t *testing.T) {
+ tests := []struct {
+ name string
+ conditionBuilder LogicalIndexedCondition
+ positiveItems []Item
+ negativeItems []Item
+ }{
+ {
+ name: "AND Two",
+ conditionBuilder: AndConditions(&mockConditionFactory{
+ list: roaring.NewRange(0, 1000),
+ }, &mockConditionFactory{
+ list: roaring.NewRange(999, 2000),
+ }),
+ positiveItems: []Item{newItem(999)},
+ negativeItems: []Item{newItem(0), newItem(1000)},
+ },
+ {
+ name: "AND Multiple Not Empty Intersection",
+ conditionBuilder: AndConditions(&mockConditionFactory{
+ list: roaring.NewRange(0, 100),
+ }, &mockConditionFactory{
+ list: roaring.NewRange(50, 150),
+ }, &mockConditionFactory{
+ list: roaring.NewRange(75, 200),
+ }),
+ positiveItems: []Item{newItem(75), newItem(99)},
+ negativeItems: []Item{newItem(49), newItem(100)},
+ },
+ {
+ name: "AND Multiple Empty Intersection",
+ conditionBuilder: AndConditions(&mockConditionFactory{
+ list: roaring.NewRange(0, 100),
+ }, &mockConditionFactory{
+ list: roaring.NewRange(100, 200),
+ }, &mockConditionFactory{
+ list: roaring.NewRange(200, 300),
+ }),
+ positiveItems: []Item{},
+ negativeItems: []Item{newItem(0), newItem(50), newItem(100), newItem(150), newItem(200)},
+ },
+ {
+ name: "OR Two",
+ conditionBuilder: OrConditions(&mockConditionFactory{
+ list: roaring.NewRange(0, 10),
+ }, &mockConditionFactory{
+ list: roaring.NewRange(90, 100),
+ }),
+ positiveItems: []Item{newItem(0), newItem(99)},
+ negativeItems: []Item{newItem(10), newItem(100)},
+ },
+ }
+
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ assert := assert.New(t)
+ filterFunc, err := tt.conditionBuilder.build(nil, nil)
+ assert.NoError(err)
+ assert.NotNil(filterFunc)
+ for _, pi := range tt.positiveItems {
+ assert.True(filterFunc.Predicate(pi))
+ }
+ for _, ni := range tt.negativeItems {
+ assert.False(filterFunc.Predicate(ni))
+ }
+ })
+ }
+}
diff --git a/banyand/tsdb/series_seek.go b/banyand/tsdb/series_seek.go
index 9051ed4..dab0917 100644
--- a/banyand/tsdb/series_seek.go
+++ b/banyand/tsdb/series_seek.go
@@ -18,17 +18,22 @@
package tsdb
import (
+ "io"
+
+ "github.com/pkg/errors"
+
"github.com/apache/skywalking-banyandb/api/common"
databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
"github.com/apache/skywalking-banyandb/banyand/kv"
"github.com/apache/skywalking-banyandb/pkg/index"
+ "github.com/apache/skywalking-banyandb/pkg/index/posting"
+ batchApi "github.com/apache/skywalking-banyandb/pkg/iter"
)
-type Iterator interface {
- Next() bool
- Val() Item
- Close() error
+type ItemIterator interface {
+ batchApi.Iterator[Item]
+ io.Closer
}
type Item interface {
@@ -47,7 +52,7 @@ type SeekerBuilder interface {
}
type Seeker interface {
- Seek() ([]Iterator, error)
+ Seek() ([]ItemIterator, error)
}
var _ SeekerBuilder = (*seekerBuilder)(nil)
@@ -55,11 +60,7 @@ var _ SeekerBuilder = (*seekerBuilder)(nil)
type seekerBuilder struct {
seriesSpan *seriesSpan
- conditions []struct {
- indexRuleType databasev1.IndexRule_Type
- indexRuleID uint32
- condition Condition
- }
+ conditions []indexedCondition
order modelv1.Sort
indexRuleForSorting *databasev1.IndexRule
rangeOptsForSorting index.RangeOpts
@@ -89,14 +90,14 @@ func newSeekerBuilder(s *seriesSpan) SeekerBuilder {
var _ Seeker = (*seeker)(nil)
type seeker struct {
- series []Iterator
+ series []ItemIterator
}
-func (s *seeker) Seek() ([]Iterator, error) {
+func (s *seeker) Seek() ([]ItemIterator, error) {
return s.series, nil
}
-func newSeeker(series []Iterator) Seeker {
+func newSeeker(series []ItemIterator) Seeker {
return &seeker{
series: series,
}
@@ -137,3 +138,28 @@ func (i *item) Val() ([]byte, error) {
func (i *item) ID() common.ItemID {
return i.itemID
}
+
+func buildSingleFilterSet(searcher index.Searcher, cond index.Condition,
+ seriesID common.SeriesID, indexRuleID uint32,
+) (posting.List, bool, error) {
+ tree, err := index.BuildTree(searcher, cond)
+ if err != nil {
+ return nil, false, err
+ }
+ _, _ = tree.TrimRangeLeaf(index.FieldKey{
+ SeriesID: seriesID,
+ IndexRuleID: indexRuleID,
+ })
+ //if found {
+ // // TODO: set sorting?
+ // //s.rangeOptsForSorting = rangeOpts
+ //}
+ itemIDs, err := tree.Execute()
+ if errors.Is(err, index.ErrEmptyTree) {
+ return nil, false, nil
+ }
+ if err != nil {
+ return nil, false, err
+ }
+ return itemIDs, true, nil
+}
diff --git a/banyand/tsdb/series_seek_filter.go b/banyand/tsdb/series_seek_filter.go
index 6a63293..e10f88a 100644
--- a/banyand/tsdb/series_seek_filter.go
+++ b/banyand/tsdb/series_seek_filter.go
@@ -25,6 +25,13 @@ import (
"github.com/apache/skywalking-banyandb/pkg/index/posting"
)
+var (
+ _ ItemFilter = (and)(nil)
+ _ ItemFilter = (or)(nil)
+ _ ItemFilter = (FilterFn)(nil)
+ _ ItemFilter = (*roaringBitmapFilter)(nil)
+)
+
var ErrUnsupportedIndexRule = errors.New("the index rule is not supported")
type Condition map[string][]index.ConditionValue
@@ -71,7 +78,7 @@ func (s *seekerBuilder) buildConditions() ([]condWithIRT, error) {
return conditions, nil
}
-func (s *seekerBuilder) buildIndexFilter(block blockDelegate, conditions []condWithIRT) (filterFn, error) {
+func (s *seekerBuilder) buildIndexFilter(block BlockDelegate, conditions []condWithIRT) (FilterFn, error) {
var allItemIDs posting.List
addIDs := func(allList posting.List, searcher index.Searcher, cond index.Condition) (posting.List, bool, error) {
tree, err := index.BuildTree(searcher, cond)
@@ -135,4 +142,4 @@ func (s *seekerBuilder) buildIndexFilter(block blockDelegate, conditions []condW
}, nil
}
-type filterFn func(item Item) bool
+type FilterFn func(item Item) bool
diff --git a/banyand/tsdb/series_seek_sort.go b/banyand/tsdb/series_seek_sort.go
index 481134e..97505b4 100644
--- a/banyand/tsdb/series_seek_sort.go
+++ b/banyand/tsdb/series_seek_sort.go
@@ -33,7 +33,7 @@ import (
"github.com/apache/skywalking-banyandb/pkg/logger"
)
-var emptyFilters = make([]filterFn, 0)
+var emptyFilters = make([]FilterFn, 0)
func (s *seekerBuilder) OrderByIndex(indexRule *databasev1.IndexRule, order modelv1.Sort) SeekerBuilder {
s.indexRuleForSorting = indexRule
@@ -47,14 +47,14 @@ func (s *seekerBuilder) OrderByTime(order modelv1.Sort) SeekerBuilder {
return s
}
-func (s *seekerBuilder) buildSeries(conditions []condWithIRT) ([]Iterator, error) {
+func (s *seekerBuilder) buildSeries(conditions []condWithIRT) ([]ItemIterator, error) {
if s.indexRuleForSorting == nil {
return s.buildSeriesByTime(conditions)
}
return s.buildSeriesByIndex(conditions)
}
-func (s *seekerBuilder) buildSeriesByIndex(conditions []condWithIRT) (series []Iterator, err error) {
+func (s *seekerBuilder) buildSeriesByIndex(conditions []condWithIRT) (series []ItemIterator, err error) {
timeFilter := func(item Item) bool {
valid := s.seriesSpan.timeRange.Contains(item.Time())
timeRange := s.seriesSpan.timeRange
@@ -70,7 +70,7 @@ func (s *seekerBuilder) buildSeriesByIndex(conditions []condWithIRT) (series []I
SeriesID: s.seriesSpan.seriesID,
IndexRuleID: s.indexRuleForSorting.GetMetadata().GetId(),
}
- filters := []filterFn{timeFilter}
+ filters := []FilterFn{timeFilter}
filter, err := s.buildIndexFilter(b, conditions)
if err != nil {
return nil, err
@@ -94,7 +94,7 @@ func (s *seekerBuilder) buildSeriesByIndex(conditions []condWithIRT) (series []I
return
}
-func (s *seekerBuilder) buildSeriesByTime(conditions []condWithIRT) ([]Iterator, error) {
+func (s *seekerBuilder) buildSeriesByTime(conditions []condWithIRT) ([]ItemIterator, error) {
bb := s.seriesSpan.blocks
switch s.order {
case modelv1.Sort_SORT_ASC, modelv1.Sort_SORT_UNSPECIFIED:
@@ -107,7 +107,7 @@ func (s *seekerBuilder) buildSeriesByTime(conditions []condWithIRT) ([]Iterator,
return bb[i].startTime().After(bb[j].startTime())
})
}
- delegated := make([]Iterator, 0, len(bb))
+ delegated := make([]ItemIterator, 0, len(bb))
bTimes := make([]time.Time, 0, len(bb))
timeRange := s.seriesSpan.timeRange
termRange := index.RangeOpts{
@@ -136,7 +136,7 @@ func (s *seekerBuilder) buildSeriesByTime(conditions []condWithIRT) ([]Iterator,
if filter == nil {
delegated = append(delegated, newSearcherIterator(s.seriesSpan.l, inner, b.dataReader(), s.seriesSpan.seriesID, emptyFilters))
} else {
- delegated = append(delegated, newSearcherIterator(s.seriesSpan.l, inner, b.dataReader(), s.seriesSpan.seriesID, []filterFn{filter}))
+ delegated = append(delegated, newSearcherIterator(s.seriesSpan.l, inner, b.dataReader(), s.seriesSpan.seriesID, []FilterFn{filter}))
}
}
}
@@ -146,10 +146,10 @@ func (s *seekerBuilder) buildSeriesByTime(conditions []condWithIRT) ([]Iterator,
Uint64("series_id", uint64(s.seriesSpan.seriesID)).
Int("shard_id", int(s.seriesSpan.shardID)).
Msg("seek series by time")
- return []Iterator{newMergedIterator(delegated)}, nil
+ return []ItemIterator{newMergedIterator(delegated)}, nil
}
-var _ Iterator = (*searcherIterator)(nil)
+var _ ItemIterator = (*searcherIterator)(nil)
type searcherIterator struct {
fieldIterator index.FieldIterator
@@ -157,11 +157,11 @@ type searcherIterator struct {
cur posting.Iterator
data kv.TimeSeriesReader
seriesID common.SeriesID
- filters []filterFn
+ filters []FilterFn
l *logger.Logger
}
-func (s *searcherIterator) Next() bool {
+func (s *searcherIterator) Next() (Item, bool) {
if s.cur == nil {
if s.fieldIterator.Next() {
v := s.fieldIterator.Val()
@@ -169,25 +169,24 @@ func (s *searcherIterator) Next() bool {
s.curKey = v.Term
s.l.Trace().Uint64("series_id", uint64(s.seriesID)).Hex("term", s.curKey).Msg("got a new field")
} else {
- return false
+ return nil, false
}
}
if s.cur.Next() {
-
for _, filter := range s.filters {
- if !filter(s.Val()) {
- s.l.Trace().Uint64("series_id", uint64(s.seriesID)).Uint64("item_id", uint64(s.Val().ID())).Msg("ignore the item")
+ if !filter(s.val()) {
+ s.l.Trace().Uint64("series_id", uint64(s.seriesID)).Uint64("item_id", uint64(s.val().ID())).Msg("ignore the item")
return s.Next()
}
}
- s.l.Trace().Uint64("series_id", uint64(s.seriesID)).Uint64("item_id", uint64(s.Val().ID())).Msg("got an item")
- return true
+ s.l.Trace().Uint64("series_id", uint64(s.seriesID)).Uint64("item_id", uint64(s.val().ID())).Msg("got an item")
+ return s.val(), true
}
s.cur = nil
return s.Next()
}
-func (s *searcherIterator) Val() Item {
+func (s *searcherIterator) val() Item {
return &item{
sortedField: s.curKey,
itemID: s.cur.Current(),
@@ -201,8 +200,8 @@ func (s *searcherIterator) Close() error {
}
func newSearcherIterator(l *logger.Logger, fieldIterator index.FieldIterator, data kv.TimeSeriesReader,
- seriesID common.SeriesID, filters []filterFn,
-) Iterator {
+ seriesID common.SeriesID, filters []FilterFn,
+) ItemIterator {
return &searcherIterator{
fieldIterator: fieldIterator,
data: data,
@@ -212,32 +211,28 @@ func newSearcherIterator(l *logger.Logger, fieldIterator index.FieldIterator, da
}
}
-var _ Iterator = (*mergedIterator)(nil)
+var _ ItemIterator = (*mergedIterator)(nil)
type mergedIterator struct {
- curr Iterator
+ curr ItemIterator
index int
- delegated []Iterator
+ delegated []ItemIterator
}
-func (m *mergedIterator) Next() bool {
+func (m *mergedIterator) Next() (Item, bool) {
if m.curr == nil {
m.index++
if m.index >= len(m.delegated) {
- return false
+ return nil, false
}
m.curr = m.delegated[m.index]
}
- hasNext := m.curr.Next()
+ nextItem, hasNext := m.curr.Next()
if !hasNext {
m.curr = nil
return m.Next()
}
- return true
-}
-
-func (m *mergedIterator) Val() Item {
- return m.curr.Val()
+ return nextItem, true
}
func (m *mergedIterator) Close() error {
@@ -248,7 +243,7 @@ func (m *mergedIterator) Close() error {
return err
}
-func newMergedIterator(delegated []Iterator) Iterator {
+func newMergedIterator(delegated []ItemIterator) ItemIterator {
return &mergedIterator{
index: -1,
delegated: delegated,
diff --git a/banyand/tsdb/series_write.go b/banyand/tsdb/series_write.go
index e3e4a22..794165f 100644
--- a/banyand/tsdb/series_write.go
+++ b/banyand/tsdb/series_write.go
@@ -47,7 +47,7 @@ var _ WriterBuilder = (*writerBuilder)(nil)
type writerBuilder struct {
series *seriesSpan
- block blockDelegate
+ block BlockDelegate
values []struct {
family []byte
val []byte
@@ -132,7 +132,7 @@ func newWriterBuilder(seriesSpan *seriesSpan) WriterBuilder {
var _ Writer = (*writer)(nil)
type writer struct {
- block blockDelegate
+ block BlockDelegate
ts time.Time
columns []struct {
family []byte
diff --git a/banyand/tsdb/seriesdb.go b/banyand/tsdb/seriesdb.go
index 9ad7cbe..4612823 100644
--- a/banyand/tsdb/seriesdb.go
+++ b/banyand/tsdb/seriesdb.go
@@ -106,18 +106,18 @@ func (p *Path) extractPrefix() {
}
}
-func (p Path) Prepand(entry Entry) Path {
+func (p Path) Prepend(entry Entry) Path {
e := Hash(entry)
- prepand := func(src []byte, entry []byte) []byte {
+ prependFunc := func(src []byte, entry []byte) []byte {
dst := make([]byte, len(src)+len(entry))
copy(dst, entry)
copy(dst[len(entry):], src)
return dst
}
- p.template = prepand(p.template, e)
+ p.template = prependFunc(p.template, e)
p.offset += len(e)
p.extractPrefix()
- p.mask = prepand(p.mask, maxIntBytes)
+ p.mask = prependFunc(p.mask, maxIntBytes)
return p
}
@@ -132,8 +132,8 @@ type SeriesDatabase interface {
type blockDatabase interface {
shardID() common.ShardID
- span(timeRange timestamp.TimeRange) ([]blockDelegate, error)
- block(id GlobalItemID) (blockDelegate, error)
+ span(timeRange timestamp.TimeRange) ([]BlockDelegate, error)
+ block(id GlobalItemID) (BlockDelegate, error)
}
var (
@@ -172,7 +172,7 @@ func (s *seriesDB) GetByID(id common.SeriesID) (Series, error) {
return newSeries(s.context(), id, s), nil
}
-func (s *seriesDB) block(id GlobalItemID) (blockDelegate, error) {
+func (s *seriesDB) block(id GlobalItemID) (BlockDelegate, error) {
seg := s.segCtrl.get(id.segID)
if seg == nil {
return nil, nil
@@ -234,9 +234,9 @@ func (s *seriesDB) List(path Path) (SeriesList, error) {
return result, err
}
-func (s *seriesDB) span(timeRange timestamp.TimeRange) ([]blockDelegate, error) {
+func (s *seriesDB) span(timeRange timestamp.TimeRange) ([]BlockDelegate, error) {
// TODO: return correct blocks
- result := make([]blockDelegate, 0)
+ result := make([]BlockDelegate, 0)
for _, s := range s.segCtrl.span(timeRange) {
dd, err := s.blockController.span(timeRange)
if err != nil {
@@ -291,8 +291,11 @@ func newSeriesDataBase(ctx context.Context, shardID common.ShardID, path string,
return sdb, nil
}
+// HashEntity runs hash function (e.g. with xxhash algorithm) on each segment of the Entity,
+// and concatenates all uint64 in byte array. So the return length of the byte array will be
+// 8 (every uint64 has 8 bytes) * length of the input entity.
func HashEntity(entity Entity) []byte {
- result := make(Entry, 0, len(entity)*8)
+ result := make([]byte, 0, len(entity)*8)
for _, entry := range entity {
result = append(result, Hash(entry)...)
}
@@ -300,7 +303,7 @@ func HashEntity(entity Entity) []byte {
}
func SeriesID(entity Entity) common.SeriesID {
- return common.SeriesID(convert.Hash((HashEntity(entity))))
+ return common.SeriesID(convert.Hash(HashEntity(entity)))
}
func Hash(entry []byte) []byte {
diff --git a/banyand/tsdb/seriesdb_test.go b/banyand/tsdb/seriesdb_test.go
index f7163dc..f5b1875 100644
--- a/banyand/tsdb/seriesdb_test.go
+++ b/banyand/tsdb/seriesdb_test.go
@@ -316,7 +316,7 @@ func TestNewPath(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
got := NewPath(tt.entity)
if tt.scope != nil {
- got = got.Prepand(tt.scope)
+ got = got.Prepend(tt.scope)
}
tester.Equal(tt.want, got)
})
diff --git a/dist/LICENSE b/dist/LICENSE
index c9abe76..1f5b378 100644
--- a/dist/LICENSE
+++ b/dist/LICENSE
@@ -247,6 +247,12 @@ BSD-2-Clause licenses
github.com/magiconair/properties v1.8.6 BSD-2-Clause
github.com/pkg/errors v0.9.1 BSD-2-Clause
+========================================================================
+BSD-2-Clause and ISC licenses
+========================================================================
+
+ github.com/emirpasic/gods v1.18.1 BSD-2-Clause and ISC
+
========================================================================
BSD-3-Clause licenses
========================================================================
@@ -317,7 +323,7 @@ MIT licenses
github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 MIT
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 MIT
go.etcd.io/bbolt v1.3.6 MIT
- go.uber.org/atomic v1.7.0 MIT
+ go.uber.org/atomic v1.9.0 MIT
go.uber.org/multierr v1.8.0 MIT
go.uber.org/zap v1.17.0 MIT
gopkg.in/natefinch/lumberjack.v2 v2.0.0 MIT
diff --git a/dist/licenses/license-github.com-emirpasic-gods.txt b/dist/licenses/license-github.com-emirpasic-gods.txt
new file mode 100644
index 0000000..e5e449b
--- /dev/null
+++ b/dist/licenses/license-github.com-emirpasic-gods.txt
@@ -0,0 +1,41 @@
+Copyright (c) 2015, Emir Pasic
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+-------------------------------------------------------------------------------
+
+AVL Tree:
+
+Copyright (c) 2017 Benjamin Scher Purcell <be...@gmail.com>
+
+Permission to use, copy, modify, and distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
diff --git a/go.mod b/go.mod
index 116eb9b..a2f5351 100644
--- a/go.mod
+++ b/go.mod
@@ -9,6 +9,7 @@ require (
github.com/cespare/xxhash v1.1.0
github.com/dgraph-io/badger/v3 v3.2011.1
github.com/dgraph-io/ristretto v0.1.0
+ github.com/emirpasic/gods v1.18.1
github.com/envoyproxy/protoc-gen-validate v0.1.0
github.com/go-chi/chi/v5 v5.0.7
github.com/golang/mock v1.6.0
@@ -111,7 +112,7 @@ require (
go.opentelemetry.io/otel/sdk/metric v0.20.0 // indirect
go.opentelemetry.io/otel/trace v0.20.0 // indirect
go.opentelemetry.io/proto/otlp v0.7.0 // indirect
- go.uber.org/atomic v1.7.0 // indirect
+ go.uber.org/atomic v1.9.0 // indirect
go.uber.org/zap v1.17.0 // indirect
golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 // indirect
golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2 // indirect
diff --git a/go.sum b/go.sum
index b62e59b..e945dff 100644
--- a/go.sum
+++ b/go.sum
@@ -153,6 +153,8 @@ github.com/dgryski/go-metro v0.0.0-20180109044635-280f6062b5bc/go.mod h1:c9O8+fp
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
+github.com/emirpasic/gods v1.18.1 h1:FXtiHYKDGKCW2KzwZKx0iC0PQmdlorYgdFG9jPXJ1Bc=
+github.com/emirpasic/gods v1.18.1/go.mod h1:8tpGGwCnJ5H4r6BWwaV6OrWmMoPhUl5jm/FMNAnJvWQ=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
@@ -548,8 +550,9 @@ go.opentelemetry.io/otel/trace v0.20.0/go.mod h1:6GjCW8zgDjwGHGa6GkyeB8+/5vjT16g
go.opentelemetry.io/proto/otlp v0.7.0 h1:rwOQPCuKAKmwGKq2aVNnYIibI6wnV7EvzgfTCzcdGg8=
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
-go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw=
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
+go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
+go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
go.uber.org/goleak v1.1.10 h1:z+mqJhf6ss6BSfSM671tgKyZBFPTTJM+HLxnhPC3wu0=
go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
diff --git a/pkg/flow/dedup_priority_queue.go b/pkg/flow/dedup_priority_queue.go
new file mode 100644
index 0000000..2a7bece
--- /dev/null
+++ b/pkg/flow/dedup_priority_queue.go
@@ -0,0 +1,136 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package flow
+
+import (
+ "container/heap"
+
+ "github.com/pkg/errors"
+)
+
+var _ heap.Interface = (*DedupPriorityQueue)(nil)
+
+// Element represents an item in the DedupPriorityQueue.
+type Element interface {
+ GetIndex() int
+ SetIndex(int)
+ // Compare returns positive number to indicate this object is greater than
+ // the other logic, 0 to indicate equality, and negative to indicate
+ // less than the other.
+ Compare(other Element) int
+}
+
+// DedupPriorityQueue implements heap.Interface.
+// DedupPriorityQueue is not thread-safe
+type DedupPriorityQueue struct {
+ Items []Element
+ cache map[Element]struct{}
+ allowDuplicates bool
+}
+
+func NewPriorityQueue(allowDuplicates bool) *DedupPriorityQueue {
+ return &DedupPriorityQueue{
+ Items: make([]Element, 0),
+ cache: make(map[Element]struct{}),
+ allowDuplicates: allowDuplicates,
+ }
+}
+
+func (pq *DedupPriorityQueue) initCache() error {
+ if pq.allowDuplicates || len(pq.Items) == 0 {
+ return nil
+ }
+ for _, elem := range pq.Items {
+ if _, ok := pq.cache[elem]; !ok {
+ pq.cache[elem] = struct{}{}
+ } else {
+ return errors.New("duplicated item is not allowed")
+ }
+ }
+ return nil
+}
+
+// Len returns the DedupPriorityQueue length.
+func (pq *DedupPriorityQueue) Len() int { return len(pq.Items) }
+
+// Less is the items less comparator.
+func (pq *DedupPriorityQueue) Less(i, j int) bool {
+ return pq.Items[i].Compare(pq.Items[j]) < 0
+}
+
+// Swap exchanges indexes of the items.
+func (pq *DedupPriorityQueue) Swap(i, j int) {
+ pq.Items[i], pq.Items[j] = pq.Items[j], pq.Items[i]
+ pq.Items[i].SetIndex(i)
+ pq.Items[j].SetIndex(j)
+}
+
+// Push implements heap.Interface.Push.
+// Appends an item to the DedupPriorityQueue.
+func (pq *DedupPriorityQueue) Push(x interface{}) {
+ item := x.(Element)
+ // if duplicates is not allowed
+ if !pq.allowDuplicates {
+ // use mutex to protect cache and items
+ // check existence
+ if _, ok := pq.cache[item]; ok {
+ return
+ }
+ pq.cache[item] = struct{}{}
+ }
+ n := len(pq.Items)
+ item.SetIndex(n)
+ pq.Items = append(pq.Items, item)
+}
+
+// Pop implements heap.Interface.Pop.
+// Removes and returns the Len() - 1 element.
+func (pq *DedupPriorityQueue) Pop() interface{} {
+ n := len(pq.Items)
+ item := pq.Items[n-1]
+ item.SetIndex(-1) // for safety
+ delete(pq.cache, item)
+ pq.Items = pq.Items[0 : n-1]
+ return item
+}
+
+// Peek returns the first item of the DedupPriorityQueue without removing it.
+func (pq *DedupPriorityQueue) Peek() Element {
+ if len(pq.Items) > 0 {
+ return (pq.Items)[0]
+ }
+ return nil
+}
+
+// Slice returns a sliced DedupPriorityQueue using the given bounds.
+func (pq *DedupPriorityQueue) Slice(start, end int) []Element {
+ return pq.Items[start:end]
+}
+
+func (pq *DedupPriorityQueue) WithNewItems(items []Element) (*DedupPriorityQueue, error) {
+ newPq := &DedupPriorityQueue{
+ Items: items,
+ cache: make(map[Element]struct{}),
+ allowDuplicates: pq.allowDuplicates,
+ }
+ err := newPq.initCache()
+ if err != nil {
+ return nil, err
+ }
+ return newPq, nil
+}
diff --git a/api/data/measure.go b/pkg/flow/op.go
similarity index 51%
copy from api/data/measure.go
copy to pkg/flow/op.go
index a582924..ebd9ca1 100644
--- a/api/data/measure.go
+++ b/pkg/flow/op.go
@@ -15,22 +15,32 @@
// specific language governing permissions and limitations
// under the License.
-package data
+package flow
import (
- "github.com/apache/skywalking-banyandb/api/common"
- "github.com/apache/skywalking-banyandb/pkg/bus"
+ "context"
)
-var MeasureWriteKindVersion = common.KindVersion{
- Version: "v1",
- Kind: "measure-write",
+// UnaryOperation represents user-defined unary function (i.e. Map, Filter, etc)
+type UnaryOperation[R any] interface {
+ Apply(ctx context.Context, data interface{}) R
}
-var TopicMeasureWrite = bus.UniTopic(MeasureWriteKindVersion.String())
+// UnaryFunc implements UnaryOperation as type func (context.Context, interface{})
+type UnaryFunc[R any] func(context.Context, interface{}) R
-var MeasureQueryKindVersion = common.KindVersion{
- Version: "v1",
- Kind: "measure-query",
+// Apply implements UnOperation.Apply method
+func (f UnaryFunc[R]) Apply(ctx context.Context, data interface{}) R {
+ return f(ctx, data)
+}
+
+// FilterFunc transform a function to an UnaryOperation
+func FilterFunc(filter UnaryOperation[bool]) (UnaryOperation[any], error) {
+ return UnaryFunc[any](func(ctx context.Context, payload interface{}) interface{} {
+ predicate := filter.Apply(ctx, payload)
+ if !predicate {
+ return nil
+ }
+ return payload
+ }), nil
}
-var TopicMeasureQuery = bus.BiTopic(MeasureQueryKindVersion.String())
diff --git a/pkg/flow/streaming/flow.go b/pkg/flow/streaming/flow.go
new file mode 100644
index 0000000..c6c7a82
--- /dev/null
+++ b/pkg/flow/streaming/flow.go
@@ -0,0 +1,120 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "context"
+ "time"
+
+ "go.uber.org/multierr"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+var _ flow.Flow = (*streamingFlow)(nil)
+
+type streamingFlow struct {
+ ctx context.Context
+ source flow.Source
+ sink flow.Sink
+ ops []flow.Operator
+ drain chan error
+}
+
+func New(source flow.Source) flow.Flow {
+ return &streamingFlow{
+ source: source,
+ ops: make([]flow.Operator, 0),
+ drain: make(chan error),
+ }
+}
+
+func (f *streamingFlow) init() error {
+ f.prepareContext()
+
+ return nil
+}
+
+func (f *streamingFlow) prepareContext() {
+ if f.ctx == nil {
+ f.ctx = context.TODO()
+ }
+
+ // TODO: add more runtime utilities
+}
+
+func (f *streamingFlow) To(sink flow.Sink) flow.Flow {
+ f.sink = sink
+ return f
+}
+
+func (f *streamingFlow) Close() error {
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ err := f.source.Teardown(ctx)
+ for _, op := range f.ops {
+ err = multierr.Append(err, op.Teardown(ctx))
+ }
+ defer close(f.drain)
+ return multierr.Append(err, f.sink.Teardown(ctx))
+}
+
+func (f *streamingFlow) Open() <-chan error {
+ if err := f.init(); err != nil {
+ go f.drainErr(err)
+ return f.drain
+ }
+
+ // setup sources
+ if err := f.source.Setup(f.ctx); err != nil {
+ go f.drainErr(err)
+ return f.drain
+ }
+
+ // setup all operators one by one
+ for _, op := range f.ops {
+ if err := op.Setup(f.ctx); err != nil {
+ go f.drainErr(err)
+ return f.drain
+ }
+ }
+
+ // setup sink
+ if err := f.sink.Setup(f.ctx); err != nil {
+ go f.drainErr(err)
+ return f.drain
+ }
+
+ // connect all operator and sink
+ for i := len(f.ops) - 1; i >= 0; i-- {
+ last := i == len(f.ops)-1
+ if last {
+ f.ops[i].Exec(f.sink)
+ } else {
+ f.ops[i].Exec(f.ops[i+1])
+ }
+ }
+ // finally connect sources and the first operator
+ f.source.Exec(f.ops[0])
+
+ return f.drain
+}
+
+func (f *streamingFlow) drainErr(err error) {
+ f.drain <- err
+}
diff --git a/pkg/flow/streaming/flow_sliding_window.go b/pkg/flow/streaming/flow_sliding_window.go
new file mode 100644
index 0000000..e130e7e
--- /dev/null
+++ b/pkg/flow/streaming/flow_sliding_window.go
@@ -0,0 +1,334 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "container/heap"
+ "context"
+ "math"
+ "sync"
+ "time"
+
+ "github.com/pkg/errors"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+type TriggerResult bool
+
+const (
+ FIRE TriggerResult = true
+ CONTINUE = false
+)
+
+var (
+ _ flow.Operator = (*TumblingTimeWindows)(nil)
+ _ flow.WindowAssigner = (*TumblingTimeWindows)(nil)
+ _ flow.Window = (*timeWindow)(nil)
+
+ _ TriggerContext = (*triggerContext)(nil)
+)
+
+func (f *streamingFlow) Window(w flow.WindowAssigner) flow.WindowedFlow {
+ switch v := w.(type) {
+ case *TumblingTimeWindows:
+ v.errorHandler = f.drainErr
+ f.ops = append(f.ops, v)
+ default:
+ f.drainErr(errors.New("window type is not supported"))
+ }
+
+ return &windowedFlow{
+ f: f,
+ wa: w,
+ }
+}
+
+func (s *windowedFlow) AllowedLateness(lateness time.Duration) flow.Flow {
+ switch v := s.wa.(type) {
+ case *TumblingTimeWindows:
+ v.lateness = lateness.Milliseconds()
+ default:
+ s.f.drainErr(errors.New("lateness is not supported"))
+ }
+ return s.f
+}
+
+type TumblingTimeWindows struct {
+ // internal state of the sliding time window
+ flow.ComponentState
+ // errorHandler is the error handler and set by the streamingFlow
+ errorHandler func(error)
+ // For TumblingTimeWindows
+ // Unit: Millisecond
+ size int64
+ // lateness is the maximum allowed lateness for incoming elements
+ // Unit: Millisecond
+ lateness int64
+
+ // guard snapshots
+ snapshotsMu sync.Mutex
+ snapshots map[int64]flow.AggregationOp
+ acc flow.AggregationOp
+
+ currentWatermark int64
+ // guard timerHeap
+ timerMu sync.Mutex
+ timerHeap *flow.DedupPriorityQueue
+
+ // aggregationFactory is the factory for creating aggregation operator
+ aggregationFactory flow.AggregationOpFactory
+
+ // For api.Operator
+ in chan flow.StreamRecord
+ out chan flow.StreamRecord
+ done chan struct{}
+ purgedWindow chan timeWindow
+}
+
+func (s *TumblingTimeWindows) In() chan<- flow.StreamRecord {
+ return s.in
+}
+
+func (s *TumblingTimeWindows) Out() <-chan flow.StreamRecord {
+ return s.out
+}
+
+func (s *TumblingTimeWindows) Setup(ctx context.Context) error {
+ if s.acc == nil {
+ s.acc = s.aggregationFactory()
+ }
+ // start processing
+ s.Add(1)
+ go s.receive()
+ // start emitting
+ s.Add(1)
+ go s.emit()
+
+ return nil
+}
+
+func (s *TumblingTimeWindows) emit() {
+ defer s.Done()
+ for w := range s.purgedWindow {
+ if err := func(window timeWindow) error {
+ s.snapshotsMu.Lock()
+ defer s.snapshotsMu.Unlock()
+
+ if snapshot, ok := s.snapshots[window.MaxTimestamp()]; ok {
+ if err := s.acc.Merge(snapshot); err != nil {
+ return err
+ }
+
+ s.out <- flow.NewStreamRecord(s.acc.Snapshot(), window.start)
+ }
+
+ return nil
+ }(w); err != nil {
+ s.errorHandler(err)
+ }
+ }
+ close(s.done)
+}
+
+func (s *TumblingTimeWindows) purgeOutdatedWindows() {
+ s.timerMu.Lock()
+ defer s.timerMu.Unlock()
+ for {
+ if lookAhead, ok := s.timerHeap.Peek().(*internalTimer); ok {
+ if lookAhead.triggerTimeMillis <= s.currentWatermark {
+ oldestTimer := heap.Pop(s.timerHeap).(*internalTimer)
+ s.purgedWindow <- oldestTimer.w
+ continue
+ }
+ }
+ return
+ }
+}
+
+func (s *TumblingTimeWindows) receive() {
+ defer s.Done()
+
+ for elem := range s.in {
+ assignedWindows, err := s.AssignWindows(elem.TimestampMillis())
+ if err != nil {
+ s.errorHandler(err)
+ continue
+ }
+ ctx := triggerContext{
+ delegation: s,
+ }
+ for _, w := range assignedWindows {
+ // drop if the window is late
+ if s.isWindowLate(w) {
+ continue
+ }
+ tw := w.(timeWindow)
+ ctx.window = tw
+ // add elem to the bucket
+ s.snapshotsMu.Lock()
+ if oldAggr, ok := s.snapshots[tw.MaxTimestamp()]; ok {
+ oldAggr.Add([]flow.StreamRecord{elem})
+ } else {
+ newAggr := s.aggregationFactory()
+ newAggr.Add([]flow.StreamRecord{elem})
+ s.snapshots[tw.MaxTimestamp()] = newAggr
+ }
+ s.snapshotsMu.Unlock()
+
+ result := ctx.OnElement(elem)
+ if result == FIRE {
+ s.purgedWindow <- tw
+ }
+ }
+
+ // even if the incoming elements do not follow strict order,
+ // the watermark could increase monotonically.
+ if elem.TimestampMillis() > s.currentWatermark {
+ s.currentWatermark = elem.TimestampMillis()
+
+ // TODO: add various strategies to allow lateness items, which come later than the current watermark
+ // Currently, assume the current watermark is t,
+ // then we allow lateness items coming after t-windowSize+slideSize, but before t,
+ // i.e. [t-windowSize+slideSize, t)
+ // NOTE: This call is the normal path to flush outdated windows immediately
+ // without considering lateness.
+ // The purged windows may be accumulated again due to non-zero lateness.
+ s.purgeOutdatedWindows()
+ }
+ }
+ close(s.purgedWindow)
+ <-s.done
+ close(s.out)
+}
+
+// isWindowLate checks whether this window is valid
+func (s *TumblingTimeWindows) isWindowLate(w flow.Window) bool {
+ return w.MaxTimestamp()+s.lateness <= s.currentWatermark
+}
+
+func (s *TumblingTimeWindows) Teardown(ctx context.Context) error {
+ s.Wait()
+ return nil
+}
+
+func (s *TumblingTimeWindows) Exec(downstream flow.Inlet) {
+ s.Add(1)
+ go flow.Transmit(&s.ComponentState, downstream, s)
+}
+
+func NewTumblingTimeWindows(size time.Duration) *TumblingTimeWindows {
+ return &TumblingTimeWindows{
+ size: size.Milliseconds(),
+ lateness: 0,
+ snapshots: make(map[int64]flow.AggregationOp),
+ timerHeap: flow.NewPriorityQueue(false),
+ in: make(chan flow.StreamRecord),
+ out: make(chan flow.StreamRecord),
+ done: make(chan struct{}),
+ purgedWindow: make(chan timeWindow),
+ currentWatermark: 0,
+ }
+}
+
+type timeWindow struct {
+ start int64
+ end int64
+}
+
+func (t timeWindow) MaxTimestamp() int64 {
+ return t.end - 1
+}
+
+// AssignWindows assigns windows according to the given timestamp
+func (s *TumblingTimeWindows) AssignWindows(timestamp int64) ([]flow.Window, error) {
+ if timestamp > math.MinInt64 {
+ start := getWindowStart(timestamp, s.size)
+ return []flow.Window{
+ timeWindow{
+ start: start,
+ end: start + s.size,
+ },
+ }, nil
+ }
+ return nil, errors.New("invalid timestamp from the element")
+}
+
+// getWindowStart calculates the window start for a timestamp.
+func getWindowStart(timestamp, windowSize int64) int64 {
+ remainder := timestamp % windowSize
+ return timestamp - remainder
+}
+
+// eventTimeTriggerOnElement processes element(s) with EventTimeTrigger
+func eventTimeTriggerOnElement(window timeWindow, ctx TriggerContext) TriggerResult {
+ if window.MaxTimestamp() <= ctx.GetCurrentWatermark() {
+ // if the watermark is already past the window fire immediately
+ return FIRE
+ }
+ ctx.RegisterEventTimeTimer(window.MaxTimestamp())
+ return CONTINUE
+}
+
+type TriggerContext interface {
+ GetCurrentWatermark() int64
+ RegisterEventTimeTimer(int64)
+ OnElement(flow.StreamRecord) TriggerResult
+}
+
+type triggerContext struct {
+ window timeWindow
+ delegation *TumblingTimeWindows
+}
+
+func (ctx *triggerContext) GetCurrentWatermark() int64 {
+ return ctx.delegation.currentWatermark
+}
+
+func (ctx *triggerContext) RegisterEventTimeTimer(triggerTime int64) {
+ ctx.delegation.timerMu.Lock()
+ defer ctx.delegation.timerMu.Unlock()
+ heap.Push(ctx.delegation.timerHeap, &internalTimer{
+ triggerTimeMillis: triggerTime,
+ w: ctx.window,
+ })
+}
+
+func (ctx *triggerContext) OnElement(record flow.StreamRecord) TriggerResult {
+ return eventTimeTriggerOnElement(ctx.window, ctx)
+}
+
+var _ flow.Element = (*internalTimer)(nil)
+
+type internalTimer struct {
+ w timeWindow
+ triggerTimeMillis int64
+ index int
+}
+
+func (t *internalTimer) GetIndex() int {
+ return t.index
+}
+
+func (t *internalTimer) SetIndex(idx int) {
+ t.index = idx
+}
+
+func (t *internalTimer) Compare(elem flow.Element) int {
+ return int(t.triggerTimeMillis - elem.(*internalTimer).triggerTimeMillis)
+}
diff --git a/pkg/flow/streaming/flow_sliding_window_test.go b/pkg/flow/streaming/flow_sliding_window_test.go
new file mode 100644
index 0000000..bf3fbbc
--- /dev/null
+++ b/pkg/flow/streaming/flow_sliding_window_test.go
@@ -0,0 +1,137 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "context"
+ "time"
+
+ . "github.com/onsi/ginkgo/v2"
+ . "github.com/onsi/gomega"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+ "github.com/apache/skywalking-banyandb/pkg/flow/streaming/sink"
+)
+
+var _ flow.AggregationOp = (*intSumAggregator)(nil)
+
+type intSumAggregator struct {
+ sum int
+}
+
+func (i *intSumAggregator) Add(input []flow.StreamRecord) {
+ for _, item := range input {
+ i.sum += item.Data().(int)
+ }
+}
+
+func (i *intSumAggregator) Merge(op flow.AggregationOp) error {
+ i.sum += op.Snapshot().(int)
+ return nil
+}
+
+func (i *intSumAggregator) Snapshot() interface{} {
+ return i.sum
+}
+
+var _ = Describe("Sliding Window", func() {
+ var (
+ baseTs time.Time
+ snk *sink.Slice
+ input []flow.StreamRecord
+ slidingWindows *TumblingTimeWindows
+
+ aggrFactory = func() flow.AggregationOp {
+ return &intSumAggregator{}
+ }
+ )
+
+ BeforeEach(func() {
+ baseTs = time.Now()
+ })
+
+ JustBeforeEach(func() {
+ snk = sink.NewSlice()
+
+ slidingWindows = NewTumblingTimeWindows(time.Second * 15)
+ slidingWindows.aggregationFactory = aggrFactory
+ slidingWindows.acc = aggrFactory()
+
+ Expect(slidingWindows.Setup(context.TODO())).Should(Succeed())
+ Expect(snk.Setup(context.TODO())).Should(Succeed())
+ slidingWindows.Exec(snk)
+ for _, r := range input {
+ slidingWindows.In() <- r
+ }
+ })
+
+ AfterEach(func() {
+ close(slidingWindows.in)
+ Expect(slidingWindows.Teardown(context.TODO())).Should(Succeed())
+ })
+
+ When("input a single element", func() {
+ BeforeEach(func() {
+ input = []flow.StreamRecord{
+ flow.NewStreamRecord(1, baseTs.UnixMilli()),
+ }
+ })
+
+ It("Should receive nothing", func() {
+ Eventually(func(g Gomega) {
+ g.Expect(slidingWindows.acc.Snapshot()).Should(Equal(0))
+ }).WithTimeout(10 * time.Second).Should(Succeed())
+ })
+ })
+
+ When("input two elements", func() {
+ BeforeEach(func() {
+ input = []flow.StreamRecord{
+ flow.NewStreamRecord(1, baseTs.UnixMilli()),
+ flow.NewStreamRecord(2, baseTs.Add(time.Minute*1).UnixMilli()),
+ }
+ })
+
+ It("Should trigger once", func() {
+ Eventually(func(g Gomega) {
+ g.Expect(snk.Value()).ShouldNot(BeNil())
+ g.Expect(snk.Value()).Should(HaveLen(1))
+ g.Expect(snk.Value()[0].(flow.StreamRecord).Data()).Should(BeEquivalentTo(1))
+ }).WithTimeout(10 * time.Second).Should(Succeed())
+ })
+ })
+
+ When("input three elements", func() {
+ BeforeEach(func() {
+ input = []flow.StreamRecord{
+ flow.NewStreamRecord(1, baseTs.UnixMilli()),
+ flow.NewStreamRecord(2, baseTs.Add(time.Second*30).UnixMilli()),
+ flow.NewStreamRecord(3, baseTs.Add(time.Minute*1).UnixMilli()),
+ }
+ })
+
+ It("Should trigger twice and merge", func() {
+ Eventually(func(g Gomega) {
+ g.Expect(snk.Value()).ShouldNot(BeNil())
+ g.Expect(snk.Value()).Should(HaveLen(2))
+ g.Expect(snk.Value()[0].(flow.StreamRecord).Data()).Should(BeEquivalentTo(1))
+ g.Expect(snk.Value()[1].(flow.StreamRecord).Data()).Should(BeEquivalentTo(3))
+ }).WithTimeout(10 * time.Second).Should(Succeed())
+ })
+ })
+})
diff --git a/pkg/flow/streaming/flow_test.go b/pkg/flow/streaming/flow_test.go
new file mode 100644
index 0000000..7b6baab
--- /dev/null
+++ b/pkg/flow/streaming/flow_test.go
@@ -0,0 +1,186 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "context"
+ "time"
+
+ . "github.com/onsi/ginkgo/v2"
+ . "github.com/onsi/gomega"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+ "github.com/apache/skywalking-banyandb/pkg/flow/streaming/sink"
+ flowTest "github.com/apache/skywalking-banyandb/pkg/test/flow"
+)
+
+// numberRange generates a slice with `count` number of integers starting from `begin`,
+// i.e. [begin, begin + count)
+func numberRange(begin, count int) []int {
+ result := make([]int, 0)
+ for i := 0; i < count; i++ {
+ result = append(result, begin+i)
+ }
+ return result
+}
+
+var _ = Describe("Streaming", func() {
+ var (
+ f flow.Flow
+ snk *sink.Slice
+ errCh <-chan error
+ )
+
+ AfterEach(func() {
+ Expect(f.Close()).Should(Succeed())
+ Consistently(errCh).ShouldNot(Receive())
+ })
+
+ Context("With Filter operator", func() {
+ var (
+ filter flow.UnaryFunc[bool]
+
+ input = flowTest.NewSlice(numberRange(0, 10))
+ )
+
+ JustBeforeEach(func() {
+ snk = sink.NewSlice()
+ f = New(input).
+ Filter(filter).
+ To(snk)
+ errCh = f.Open()
+ Expect(errCh).ShouldNot(BeNil())
+ })
+
+ When("Given a odd filter", func() {
+ BeforeEach(func() {
+ filter = func(ctx context.Context, i interface{}) bool {
+ return i.(int)%2 == 0
+ }
+ })
+
+ It("Should filter odd number", func() {
+ Eventually(func(g Gomega) {
+ g.Expect(snk.Value()).Should(Equal([]interface{}{
+ flow.NewStreamRecordWithoutTS(0),
+ flow.NewStreamRecordWithoutTS(2),
+ flow.NewStreamRecordWithoutTS(4),
+ flow.NewStreamRecordWithoutTS(6),
+ flow.NewStreamRecordWithoutTS(8),
+ }))
+ }).Should(Succeed())
+ })
+ })
+ })
+
+ Context("With Mapper operator", func() {
+ var (
+ mapper flow.UnaryFunc[any]
+
+ input = flowTest.NewSlice(numberRange(0, 10))
+ )
+
+ JustBeforeEach(func() {
+ snk = sink.NewSlice()
+ f = New(input).
+ Map(mapper).
+ To(snk)
+ errCh = f.Open()
+ Expect(errCh).ShouldNot(BeNil())
+ })
+
+ When("given a multiplier", func() {
+ BeforeEach(func() {
+ mapper = func(ctx context.Context, i interface{}) interface{} {
+ return i.(int) * 2
+ }
+ })
+
+ It("Should multiply by 2", func() {
+ Eventually(func(g Gomega) {
+ g.Expect(snk.Value()).Should(Equal([]interface{}{
+ flow.NewStreamRecordWithoutTS(0),
+ flow.NewStreamRecordWithoutTS(2),
+ flow.NewStreamRecordWithoutTS(4),
+ flow.NewStreamRecordWithoutTS(6),
+ flow.NewStreamRecordWithoutTS(8),
+ flow.NewStreamRecordWithoutTS(10),
+ flow.NewStreamRecordWithoutTS(12),
+ flow.NewStreamRecordWithoutTS(14),
+ flow.NewStreamRecordWithoutTS(16),
+ flow.NewStreamRecordWithoutTS(18),
+ }))
+ }).Should(Succeed())
+ })
+ })
+ })
+
+ Context("With TopN operator", func() {
+ type record struct {
+ service string
+ instance string
+ value int
+ }
+
+ var input []flow.StreamRecord
+
+ JustBeforeEach(func() {
+ snk = sink.NewSlice()
+
+ f = New(flowTest.NewSlice(input)).
+ Map(flow.UnaryFunc[any](func(ctx context.Context, item interface{}) interface{} {
+ // groupBy
+ return flow.Data{item.(*record).service, int64(item.(*record).value)}
+ })).
+ Window(NewTumblingTimeWindows(15*time.Second)).
+ TopN(3, WithSortKeyExtractor(func(record flow.StreamRecord) int64 {
+ return record.Data().(flow.Data)[1].(int64)
+ })).
+ To(snk)
+
+ errCh = f.Open()
+ Expect(errCh).ShouldNot(BeNil())
+ })
+
+ When("Top3", func() {
+ BeforeEach(func() {
+ input = []flow.StreamRecord{
+ flow.NewStreamRecord(&record{"e2e-service-provider", "instance-001", 10000}, 1000),
+ flow.NewStreamRecord(&record{"e2e-service-consumer", "instance-001", 9900}, 2000),
+ flow.NewStreamRecord(&record{"e2e-service-provider", "instance-002", 9800}, 3000),
+ flow.NewStreamRecord(&record{"e2e-service-consumer", "instance-002", 9700}, 4000),
+ flow.NewStreamRecord(&record{"e2e-service-provider", "instance-003", 9700}, 5000),
+ flow.NewStreamRecord(&record{"e2e-service-consumer", "instance-004", 9600}, 6000),
+ flow.NewStreamRecord(&record{"e2e-service-consumer", "instance-001", 9500}, 7000),
+ flow.NewStreamRecord(&record{"e2e-service-provider", "instance-002", 9800}, 61000),
+ }
+ })
+
+ It("Should take top 3 elements", func() {
+ Eventually(func(g Gomega) {
+ g.Expect(len(snk.Value())).Should(BeNumerically(">=", 1))
+ g.Expect(snk.Value()[0].(flow.StreamRecord).Data()).Should(BeEquivalentTo([]*Tuple2{
+ {int64(9500), flow.NewStreamRecord(flow.Data{"e2e-service-consumer", int64(9500)}, 7000)},
+ {int64(9600), flow.NewStreamRecord(flow.Data{"e2e-service-consumer", int64(9600)}, 6000)},
+ {int64(9700), flow.NewStreamRecord(flow.Data{"e2e-service-consumer", int64(9700)}, 4000)},
+ }))
+ }).WithTimeout(10 * time.Second).Should(Succeed())
+ })
+ })
+ })
+})
diff --git a/pkg/flow/streaming/flow_topn.go b/pkg/flow/streaming/flow_topn.go
new file mode 100644
index 0000000..79ad7e6
--- /dev/null
+++ b/pkg/flow/streaming/flow_topn.go
@@ -0,0 +1,171 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "github.com/emirpasic/gods/maps/treemap"
+ "github.com/emirpasic/gods/utils"
+ "github.com/google/go-cmp/cmp"
+ "github.com/pkg/errors"
+
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+type windowedFlow struct {
+ f *streamingFlow
+ wa flow.WindowAssigner
+}
+
+func (s *windowedFlow) TopN(topNum int, opts ...any) flow.Flow {
+ s.wa.(*TumblingTimeWindows).aggregationFactory = func() flow.AggregationOp {
+ topNAggrFunc := &topNAggregator{
+ cacheSize: topNum,
+ sort: modelv1.Sort_SORT_UNSPECIFIED,
+ }
+ // apply user customized options
+ for _, opt := range opts {
+ if applier, ok := opt.(TopNOption); ok {
+ applier(topNAggrFunc)
+ }
+ }
+ if topNAggrFunc.sortKeyExtractor == nil {
+ s.f.drainErr(errors.New("sortKeyExtractor must be specified"))
+ }
+ if topNAggrFunc.sort == modelv1.Sort_SORT_DESC {
+ topNAggrFunc.comparator = func(a, b interface{}) int {
+ return utils.Int64Comparator(b, a)
+ }
+ } else {
+ topNAggrFunc.comparator = utils.Int64Comparator
+ }
+ topNAggrFunc.treeMap = treemap.NewWith(topNAggrFunc.comparator)
+ return topNAggrFunc
+ }
+ return s.f
+}
+
+type topNAggregator struct {
+ // cacheSize is the maximum number of entries which can be held in the buffer, i.e. treeMap
+ cacheSize int
+ // currentTopNum indicates how many records are tracked.
+ // This should not exceed cacheSize
+ currentTopNum int
+ treeMap *treemap.Map
+ // sortKeyExtractor is an extractor to fetch sort key from the record
+ // TODO: currently we only support sorting numeric field, i.e. int64
+ sortKeyExtractor func(flow.StreamRecord) int64
+ // sort indicates the order of results
+ sort modelv1.Sort
+ comparator utils.Comparator
+}
+
+type TopNOption func(aggregator *topNAggregator)
+
+func WithSortKeyExtractor(sortKeyExtractor func(flow.StreamRecord) int64) TopNOption {
+ return func(aggregator *topNAggregator) {
+ aggregator.sortKeyExtractor = sortKeyExtractor
+ }
+}
+
+func OrderBy(sort modelv1.Sort) TopNOption {
+ return func(aggregator *topNAggregator) {
+ aggregator.sort = sort
+ }
+}
+
+func (t *topNAggregator) Add(input []flow.StreamRecord) {
+ for _, item := range input {
+ sortKey := t.sortKeyExtractor(item)
+ // check
+ if t.checkSortKeyInBufferRange(sortKey) {
+ t.put(sortKey, item)
+ t.doCleanUp()
+ }
+ }
+}
+
+func (t *topNAggregator) doCleanUp() {
+ // do cleanup: maintain the treeMap size
+ if t.currentTopNum > t.cacheSize {
+ lastKey, lastValues := t.treeMap.Max()
+ size := len(lastValues.([]interface{}))
+ // remove last one
+ if size <= 1 {
+ t.currentTopNum -= size
+ t.treeMap.Remove(lastKey)
+ } else {
+ t.currentTopNum--
+ t.treeMap.Put(lastKey, lastValues.([]interface{})[0:size-1])
+ }
+ }
+}
+
+func (t *topNAggregator) Merge(other flow.AggregationOp) error {
+ for _, tuple2 := range other.Snapshot().([]*Tuple2) {
+ t.put(tuple2.V1.(int64), tuple2.V2.(flow.StreamRecord))
+ }
+ t.doCleanUp()
+ return nil
+}
+
+func (t *topNAggregator) put(sortKey int64, data flow.StreamRecord) {
+ t.currentTopNum++
+ if existingList, ok := t.treeMap.Get(sortKey); ok {
+ existingList = append(existingList.([]interface{}), data)
+ t.treeMap.Put(sortKey, existingList)
+ } else {
+ t.treeMap.Put(sortKey, []interface{}{data})
+ }
+}
+
+func (t *topNAggregator) checkSortKeyInBufferRange(sortKey int64) bool {
+ // get the "maximum" item
+ // - if ASC, the maximum item
+ // - else DESC, the minimum item
+ worstKey, _ := t.treeMap.Max()
+ if worstKey == nil {
+ // return true if the buffer is empty.
+ return true
+ }
+ if t.comparator(sortKey, worstKey.(int64)) < 0 {
+ return true
+ }
+ return t.currentTopNum < t.cacheSize
+}
+
+type Tuple2 struct {
+ V1 interface{} `json:"v1"`
+ V2 interface{} `json:"v2"`
+}
+
+func (t *Tuple2) Equal(other *Tuple2) bool {
+ return cmp.Equal(t.V1, other.V1) && cmp.Equal(t.V2, other.V2)
+}
+
+func (t *topNAggregator) Snapshot() interface{} {
+ iter := t.treeMap.Iterator()
+ items := make([]*Tuple2, 0, t.currentTopNum)
+ for iter.Next() {
+ list := iter.Value().([]interface{})
+ for _, item := range list {
+ items = append(items, &Tuple2{iter.Key(), item})
+ }
+ }
+ return items
+}
diff --git a/pkg/flow/streaming/flow_topn_test.go b/pkg/flow/streaming/flow_topn_test.go
new file mode 100644
index 0000000..dbc1f84
--- /dev/null
+++ b/pkg/flow/streaming/flow_topn_test.go
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "testing"
+
+ "github.com/emirpasic/gods/maps/treemap"
+ "github.com/emirpasic/gods/utils"
+ "github.com/stretchr/testify/require"
+
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+func TestFlow_TopN_Aggregator(t *testing.T) {
+ input := []flow.StreamRecord{
+ // 1. string
+ // 2. number
+ // 3. slices of groupBy values
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-provider", 10000, []interface{}{"e2e-service-provider"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9900, []interface{}{"e2e-service-consumer"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-provider", 9800, []interface{}{"e2e-service-provider"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9700, []interface{}{"e2e-service-consumer"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-provider", 9700, []interface{}{"e2e-service-provider"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9600, []interface{}{"e2e-service-consumer"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9800, []interface{}{"e2e-service-consumer"}}),
+ flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9500, []interface{}{"e2e-service-consumer"}}),
+ }
+ tests := []struct {
+ name string
+ sort modelv1.Sort
+ expected []*Tuple2
+ }{
+ {
+ name: "DESC",
+ sort: modelv1.Sort_SORT_DESC,
+ expected: []*Tuple2{
+ {int64(10000), flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-provider", 10000, []interface{}{"e2e-service-provider"}})},
+ {int64(9900), flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9900, []interface{}{"e2e-service-consumer"}})},
+ {int64(9800), flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-provider", 9800, []interface{}{"e2e-service-provider"}})},
+ },
+ },
+ {
+ name: "ASC",
+ sort: modelv1.Sort_SORT_ASC,
+ expected: []*Tuple2{
+ {int64(9500), flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9500, []interface{}{"e2e-service-consumer"}})},
+ {int64(9600), flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9600, []interface{}{"e2e-service-consumer"}})},
+ {int64(9700), flow.NewStreamRecordWithoutTS(flow.Data{"e2e-service-consumer", 9700, []interface{}{"e2e-service-consumer"}})},
+ },
+ },
+ }
+ for _, tt := range tests {
+ t.Run(tt.name, func(t *testing.T) {
+ require := require.New(t)
+ var comparator utils.Comparator
+ if tt.sort == modelv1.Sort_SORT_DESC {
+ comparator = func(a, b interface{}) int {
+ return utils.Int64Comparator(b, a)
+ }
+ } else {
+ comparator = utils.Int64Comparator
+ }
+ topN := &topNAggregator{
+ cacheSize: 3,
+ sort: tt.sort,
+ comparator: comparator,
+ treeMap: treemap.NewWith(comparator),
+ sortKeyExtractor: func(record flow.StreamRecord) int64 {
+ return int64(record.Data().(flow.Data)[1].(int))
+ },
+ }
+ topN.Add(input)
+ require.Len(topN.Snapshot(), 3)
+ require.Equal(tt.expected, topN.Snapshot())
+ })
+ }
+}
diff --git a/pkg/flow/streaming/flow_unary.go b/pkg/flow/streaming/flow_unary.go
new file mode 100644
index 0000000..f12f2f8
--- /dev/null
+++ b/pkg/flow/streaming/flow_unary.go
@@ -0,0 +1,109 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package streaming
+
+import (
+ "context"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+func (f *streamingFlow) Filter(predicate flow.UnaryOperation[bool]) flow.Flow {
+ op, err := flow.FilterFunc(predicate)
+ if err != nil {
+ f.drainErr(err)
+ }
+ return f.Transform(op)
+}
+
+func (f *streamingFlow) Map(mapper flow.UnaryOperation[any]) flow.Flow {
+ return f.Transform(mapper)
+}
+
+// Transform represents a general unary transformation
+// For example: filter, map, etc.
+func (f *streamingFlow) Transform(op flow.UnaryOperation[any]) flow.Flow {
+ // TODO: support parallelism
+ f.ops = append(f.ops, newUnaryOp(op, 1))
+ return f
+}
+
+var _ flow.Operator = (*unaryOperator)(nil)
+
+type unaryOperator struct {
+ flow.ComponentState
+ op flow.UnaryOperation[any]
+ in chan flow.StreamRecord
+ out chan flow.StreamRecord
+ parallelism uint
+}
+
+func (u *unaryOperator) Setup(ctx context.Context) error {
+ // run a background job as a consumer
+ go u.run()
+ return nil
+}
+
+func (u *unaryOperator) Exec(downstream flow.Inlet) {
+ u.Add(1)
+ // start a background job for transmission
+ go flow.Transmit(&u.ComponentState, downstream, u)
+}
+
+func (u *unaryOperator) Teardown(ctx context.Context) error {
+ u.Wait()
+ return nil
+}
+
+func newUnaryOp(op flow.UnaryOperation[any], parallelism uint) *unaryOperator {
+ return &unaryOperator{
+ op: op,
+ in: make(chan flow.StreamRecord),
+ out: make(chan flow.StreamRecord),
+ parallelism: parallelism,
+ }
+}
+
+func (u *unaryOperator) In() chan<- flow.StreamRecord {
+ return u.in
+}
+
+func (u *unaryOperator) Out() <-chan flow.StreamRecord {
+ return u.out
+}
+
+func (u *unaryOperator) run() {
+ semaphore := make(chan struct{}, u.parallelism)
+ for elem := range u.in {
+ semaphore <- struct{}{}
+ go func(r flow.StreamRecord) {
+ defer func() { <-semaphore }()
+ result := u.op.Apply(context.TODO(), r.Data())
+ switch val := result.(type) {
+ case nil:
+ return
+ default:
+ u.out <- r.WithNewData(val)
+ }
+ }(elem)
+ }
+ for i := 0; i < int(u.parallelism); i++ {
+ semaphore <- struct{}{}
+ }
+ close(u.out)
+}
diff --git a/pkg/flow/streaming/sink/slice.go b/pkg/flow/streaming/sink/slice.go
new file mode 100644
index 0000000..ad55850
--- /dev/null
+++ b/pkg/flow/streaming/sink/slice.go
@@ -0,0 +1,82 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package sink
+
+import (
+ "context"
+ "sync"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+var _ flow.Sink = (*Slice)(nil)
+
+type Slice struct {
+ sync.RWMutex
+ flow.ComponentState
+ slice []interface{}
+ in chan flow.StreamRecord
+}
+
+func NewSlice() *Slice {
+ return &Slice{
+ slice: make([]interface{}, 0),
+ in: make(chan flow.StreamRecord),
+ }
+}
+
+func (s *Slice) Value() []interface{} {
+ s.RLock()
+ defer s.RUnlock()
+ return s.slice
+}
+
+func (s *Slice) In() chan<- flow.StreamRecord {
+ return s.in
+}
+
+func (s *Slice) Setup(ctx context.Context) error {
+ go s.run(ctx)
+
+ return nil
+}
+
+func (s *Slice) run(ctx context.Context) {
+ s.Add(1)
+ defer func() {
+ s.Done()
+ }()
+ for {
+ select {
+ case item, ok := <-s.in:
+ if !ok {
+ return
+ }
+ s.Lock()
+ s.slice = append(s.slice, item)
+ s.Unlock()
+ case <-ctx.Done():
+ return
+ }
+ }
+}
+
+func (s *Slice) Teardown(ctx context.Context) error {
+ s.Wait()
+ return nil
+}
diff --git a/pkg/flow/streaming/sources/channel.go b/pkg/flow/streaming/sources/channel.go
new file mode 100644
index 0000000..eb71354
--- /dev/null
+++ b/pkg/flow/streaming/sources/channel.go
@@ -0,0 +1,93 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package sources
+
+import (
+ "context"
+ "reflect"
+
+ "github.com/pkg/errors"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+var _ flow.Source = (*sourceChan)(nil)
+
+type sourceChan struct {
+ flow.ComponentState
+ in interface{}
+ out chan flow.StreamRecord
+}
+
+func (s *sourceChan) Out() <-chan flow.StreamRecord {
+ return s.out
+}
+
+func (s *sourceChan) Setup(ctx context.Context) error {
+ chanVal := reflect.ValueOf(s.in)
+
+ if !chanVal.IsValid() {
+ return errors.New("invalid channel")
+ }
+
+ s.Add(1)
+ go s.run(ctx, chanVal)
+ return nil
+}
+
+func (s *sourceChan) run(ctx context.Context, chanVal reflect.Value) {
+ ctx, cancel := context.WithCancel(ctx)
+ defer func() {
+ cancel()
+ close(s.out)
+ s.Done()
+ }()
+
+ for {
+ val, open := chanVal.Recv()
+ if !open {
+ return
+ }
+ select {
+ case s.out <- flow.TryExactTimestamp(val.Interface()):
+ case <-ctx.Done():
+ return
+ }
+ }
+}
+
+func (s *sourceChan) Teardown(ctx context.Context) error {
+ s.Wait()
+ return nil
+}
+
+func (s *sourceChan) Exec(downstream flow.Inlet) {
+ s.Add(1)
+ go flow.Transmit(&s.ComponentState, downstream, s)
+}
+
+func NewChannel(in interface{}) (flow.Source, error) {
+ if reflect.TypeOf(in).Kind() != reflect.Chan {
+ return nil, errors.New("in must be a Channel")
+ }
+
+ return &sourceChan{
+ in: in,
+ out: make(chan flow.StreamRecord, 1024),
+ }, nil
+}
diff --git a/pkg/flow/streaming/sources/channel_test.go b/pkg/flow/streaming/sources/channel_test.go
new file mode 100644
index 0000000..984d513
--- /dev/null
+++ b/pkg/flow/streaming/sources/channel_test.go
@@ -0,0 +1,72 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package sources_test
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/golang/mock/gomock"
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+ "github.com/apache/skywalking-banyandb/pkg/flow/streaming/sources"
+)
+
+const (
+ ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+)
+
+func TestSource_channel(t *testing.T) {
+ ctrl := gomock.NewController(t)
+ defer ctrl.Finish()
+
+ inlet := flow.NewMockInlet(ctrl)
+
+ srcCh := make(chan string)
+
+ assert := require.New(t)
+ src, err := sources.NewChannel(srcCh)
+ assert.NoError(err)
+ assert.NoError(src.Setup(context.TODO()))
+
+ in := make(chan flow.StreamRecord)
+ inlet.
+ EXPECT().
+ In().Times(len(ALPHABET) + 1).
+ Return(in)
+
+ src.Exec(inlet)
+
+ go func() {
+ for _, char := range strings.Split(ALPHABET, "") {
+ srcCh <- char
+ }
+ close(srcCh)
+ }()
+
+ var result strings.Builder
+ for item := range in {
+ assert.IsType(flow.StreamRecord{}, item)
+ result.WriteString(item.Data().(string))
+ }
+
+ assert.Equal(ALPHABET, result.String())
+ assert.NoError(src.Teardown(context.TODO()))
+}
diff --git a/api/data/measure.go b/pkg/flow/streaming/streaming_suite_test.go
similarity index 65%
copy from api/data/measure.go
copy to pkg/flow/streaming/streaming_suite_test.go
index a582924..746890d 100644
--- a/api/data/measure.go
+++ b/pkg/flow/streaming/streaming_suite_test.go
@@ -15,22 +15,16 @@
// specific language governing permissions and limitations
// under the License.
-package data
+package streaming_test
import (
- "github.com/apache/skywalking-banyandb/api/common"
- "github.com/apache/skywalking-banyandb/pkg/bus"
-)
-
-var MeasureWriteKindVersion = common.KindVersion{
- Version: "v1",
- Kind: "measure-write",
-}
+ "testing"
-var TopicMeasureWrite = bus.UniTopic(MeasureWriteKindVersion.String())
+ . "github.com/onsi/ginkgo/v2"
+ . "github.com/onsi/gomega"
+)
-var MeasureQueryKindVersion = common.KindVersion{
- Version: "v1",
- Kind: "measure-query",
+func TestStreaming(t *testing.T) {
+ RegisterFailHandler(Fail)
+ RunSpecs(t, "Streaming Suite")
}
-var TopicMeasureQuery = bus.BiTopic(MeasureQueryKindVersion.String())
diff --git a/pkg/flow/types.go b/pkg/flow/types.go
new file mode 100644
index 0000000..7b025b2
--- /dev/null
+++ b/pkg/flow/types.go
@@ -0,0 +1,174 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package flow
+
+import (
+ "context"
+ "io"
+ "sync"
+ "time"
+
+ "google.golang.org/protobuf/types/known/timestamppb"
+)
+
+type Data []any
+
+// Flow is an abstraction of data flow for
+// both Streaming and Batch
+type Flow interface {
+ io.Closer
+ // Filter is used to filter data.
+ // The parameter f can be either predicate function for streaming,
+ // or conditions for batch query.
+ Filter(UnaryOperation[bool]) Flow
+ // Map is used to transform data
+ Map(UnaryOperation[any]) Flow
+ // Window is used to split infinite data into "buckets" of finite size.
+ // Currently, it is only applicable to streaming context.
+ Window(WindowAssigner) WindowedFlow
+ // To pipes data to the given sink
+ To(sink Sink) Flow
+ // Open opens the flow in the async mode for streaming scenario.
+ // The first error is the error combination while opening all components,
+ // while the second is a channel for receiving async errors.
+ Open() <-chan error
+}
+
+// WindowedFlow is a flow which processes incoming elements based on window.
+// The WindowedFlow can be created with a WindowAssigner.
+type WindowedFlow interface {
+ AllowedLateness(lateness time.Duration) Flow
+ // TopN applies a TopNAggregation to each Window.
+ TopN(topNum int, opts ...any) Flow
+}
+
+// Window is a bucket of elements with a finite size.
+// timedWindow is the only implementation now.
+type Window interface {
+ // MaxTimestamp returns the upper bound of the Window.
+ // Unit: Millisecond
+ MaxTimestamp() int64
+}
+
+// WindowAssigner is used to assign Window(s) for a given timestamp, and thus it can create a WindowedFlow.
+type WindowAssigner interface {
+ // AssignWindows assigns a slice of Window according to the given timestamp, e.g. eventTime.
+ // The unit of the timestamp here is MilliSecond.
+ AssignWindows(timestamp int64) ([]Window, error)
+}
+
+// AggregationOp defines the stateful operation for aggregation.
+type AggregationOp interface {
+ // Add puts a slice of elements as the input
+ Add([]StreamRecord)
+ // Merge merges the argument with the current Op
+ Merge(AggregationOp) error
+ // Snapshot takes a snapshot of the current state of the AggregationOp
+ Snapshot() interface{}
+}
+
+type AggregationOpFactory func() AggregationOp
+
+// StreamRecord is a container wraps user data and timestamp.
+// It is the underlying transmission medium for the streaming processing.
+type StreamRecord struct {
+ ts int64
+ data interface{}
+}
+
+func NewStreamRecord(data interface{}, ts int64) StreamRecord {
+ return StreamRecord{
+ data: data,
+ ts: ts,
+ }
+}
+
+func NewStreamRecordWithTimestampPb(data interface{}, timestamp *timestamppb.Timestamp) StreamRecord {
+ return StreamRecord{
+ data: data,
+ ts: timestamp.GetSeconds()*1000 + int64(timestamp.GetNanos())/1000_000,
+ }
+}
+
+func NewStreamRecordWithoutTS(data interface{}) StreamRecord {
+ return StreamRecord{
+ data: data,
+ ts: -1,
+ }
+}
+
+func (sr StreamRecord) WithNewData(data interface{}) StreamRecord {
+ return StreamRecord{
+ ts: sr.ts,
+ data: data,
+ }
+}
+
+func (sr StreamRecord) TimestampMillis() int64 {
+ return sr.ts
+}
+
+func (sr StreamRecord) Data() interface{} {
+ return sr.data
+}
+
+// Inlet represents a type that exposes one open input.
+//
+//go:generate mockgen -destination=./inlet_mock.go -package=flow github.com/apache/skywalking-banyandb/pkg/flow Inlet
+type Inlet interface {
+ In() chan<- StreamRecord
+}
+
+// Outlet represents a type that exposes one open output.
+type Outlet interface {
+ Out() <-chan StreamRecord
+}
+
+type Component interface {
+ // Setup is the lifecycle hook for resource preparation, e.g. start background job for listening input channel.
+ // It must be called before the flow starts to process elements.
+ Setup(context.Context) error
+ // Teardown is the lifecycle hook for shutting down the Component
+ // Implementation should ENSURE that all resource has been correctly recycled before this method returns.
+ Teardown(context.Context) error
+}
+
+type ComponentState struct {
+ sync.WaitGroup
+}
+
+// Source represents a set of stream processing steps that has one open output.
+type Source interface {
+ Outlet
+ Component
+ Exec(downstream Inlet)
+}
+
+// Operator represents a set of stream processing steps that has one open input and one open output.
+type Operator interface {
+ Inlet
+ Outlet
+ Component
+ Exec(downstream Inlet)
+}
+
+// Sink represents a set of stream processing steps that has one open input.
+type Sink interface {
+ Inlet
+ Component
+}
diff --git a/api/data/measure.go b/pkg/flow/utils.go
similarity index 53%
copy from api/data/measure.go
copy to pkg/flow/utils.go
index a582924..7f4d805 100644
--- a/api/data/measure.go
+++ b/pkg/flow/utils.go
@@ -15,22 +15,28 @@
// specific language governing permissions and limitations
// under the License.
-package data
+package flow
-import (
- "github.com/apache/skywalking-banyandb/api/common"
- "github.com/apache/skywalking-banyandb/pkg/bus"
-)
-
-var MeasureWriteKindVersion = common.KindVersion{
- Version: "v1",
- Kind: "measure-write",
+// Transmit provides a helper function to connect the current component with the downstream component.
+// It should be run in another goroutine.
+func Transmit(state *ComponentState, downstream Inlet, current Outlet) {
+ defer state.Done()
+ for elem := range current.Out() {
+ downstream.In() <- elem
+ }
+ close(downstream.In())
}
-var TopicMeasureWrite = bus.UniTopic(MeasureWriteKindVersion.String())
-
-var MeasureQueryKindVersion = common.KindVersion{
- Version: "v1",
- Kind: "measure-query",
+func TryExactTimestamp(item any) StreamRecord {
+ if r, ok := item.(StreamRecord); ok {
+ return r
+ }
+ type timestampExtractor interface {
+ TimestampMillis() int64
+ }
+ // otherwise, check if we can extract timestamp
+ if extractor, ok := item.(timestampExtractor); ok {
+ return NewStreamRecord(item, extractor.TimestampMillis())
+ }
+ return NewStreamRecordWithoutTS(item)
}
-var TopicMeasureQuery = bus.BiTopic(MeasureQueryKindVersion.String())
diff --git a/pkg/iter/iter.go b/pkg/iter/iter.go
new file mode 100644
index 0000000..d2406c6
--- /dev/null
+++ b/pkg/iter/iter.go
@@ -0,0 +1,109 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package iter
+
+// An Iterator is a stream of items of some type.
+type Iterator[T any] interface {
+ // Next checks whether if the iteration has more elements and
+ // returns the next one if exists.
+ Next() (T, bool)
+}
+
+// FromSlice creates a new iterator which returns all items from the slice starting at index 0 until
+// all items are consumed.
+func FromSlice[T any](slice []T) Iterator[T] {
+ return &sliceIterator[T]{slice: slice}
+}
+
+type sliceIterator[T any] struct {
+ slice []T
+}
+
+func (iter *sliceIterator[T]) Next() (T, bool) {
+ if len(iter.slice) == 0 {
+ var zero T
+ return zero, false
+ }
+ item := iter.slice[0]
+ iter.slice = iter.slice[1:]
+ return item, true
+}
+
+// Map returns a new iterator which applies a function to all items from the input iterator which
+// are subsequently returned.
+//
+// The mapping function should not mutate the state outside its scope.
+func Map[T any, O any](from Iterator[T], mapFunc func(T) O) Iterator[O] {
+ return &mapIterator[T, O]{from: from, mapFunc: mapFunc}
+}
+
+type mapIterator[T any, O any] struct {
+ from Iterator[T]
+ mapFunc func(T) O
+}
+
+func (iter *mapIterator[T, O]) Next() (O, bool) {
+ item, ok := iter.from.Next()
+ if !ok {
+ var zero O
+ return zero, false
+ }
+ mapped := iter.mapFunc(item)
+ return mapped, true
+}
+
+// Flatten applies a function to all items of the specified iterator, returning an iterator for each
+// item. The resulting iterators are then concatenated into a single iterator.
+func Flatten[T any](from Iterator[Iterator[T]]) Iterator[T] {
+ return &flattenIterator[T]{from: from}
+}
+
+type flattenIterator[T any] struct {
+ from Iterator[Iterator[T]]
+ head Iterator[T]
+}
+
+func (iter *flattenIterator[T]) Next() (T, bool) {
+ for {
+ if iter.head == nil {
+ item, ok := iter.from.Next()
+ if !ok {
+ var zero T
+ return zero, false
+ }
+ iter.head = item
+ }
+ item, ok := iter.head.Next()
+ if ok {
+ return item, true
+ }
+ iter.head = nil
+ }
+}
+
+// Empty returns an iterator that never returns anything.
+func Empty[T any]() Iterator[T] {
+ return emptyIterator[T]{}
+}
+
+type emptyIterator[T any] struct{}
+
+func (emptyIterator[T]) Next() (T, bool) {
+ var zero T
+ return zero, false
+}
diff --git a/pkg/pb/v1/query.go b/pkg/pb/v1/query.go
index b3e085b..86cf507 100644
--- a/pkg/pb/v1/query.go
+++ b/pkg/pb/v1/query.go
@@ -366,3 +366,111 @@ func (b *GroupByBuilder) aggregate(fieldName string, aggregationFunc modelv1.Agg
}
return b.builder
}
+
+type MeasureTopNRequestBuilder struct {
+ req *measurev1.TopNRequest
+}
+
+func NewMeasureTopNRequestBuilder() *MeasureTopNRequestBuilder {
+ return &MeasureTopNRequestBuilder{
+ req: &measurev1.TopNRequest{},
+ }
+}
+
+func (b *MeasureTopNRequestBuilder) Metadata(group, name string) *MeasureTopNRequestBuilder {
+ b.req.Metadata = &commonv1.Metadata{
+ Group: group,
+ Name: name,
+ }
+ return b
+}
+
+func (b *MeasureTopNRequestBuilder) TimeRange(sT, eT time.Time) *MeasureTopNRequestBuilder {
+ b.req.TimeRange = &modelv1.TimeRange{
+ Begin: timestamppb.New(sT),
+ End: timestamppb.New(eT),
+ }
+ return b
+}
+
+func (b *MeasureTopNRequestBuilder) Conditions(conditions ...*modelv1.Condition) *MeasureTopNRequestBuilder {
+ b.req.Conditions = conditions
+ return b
+}
+
+func (b *MeasureTopNRequestBuilder) TopN(num int32) *MeasureTopNRequestBuilder {
+ b.req.TopN = num
+ return b
+}
+
+func (b *MeasureTopNRequestBuilder) Max() *MeasureTopNRequestBuilder {
+ return b.Aggregate(modelv1.AggregationFunction_AGGREGATION_FUNCTION_MAX)
+}
+
+func (b *MeasureTopNRequestBuilder) Min() *MeasureTopNRequestBuilder {
+ return b.Aggregate(modelv1.AggregationFunction_AGGREGATION_FUNCTION_MIN)
+}
+
+func (b *MeasureTopNRequestBuilder) Aggregate(aggMethod modelv1.AggregationFunction) *MeasureTopNRequestBuilder {
+ b.req.Agg = aggMethod
+ return b
+}
+
+func (b *MeasureTopNRequestBuilder) Build() *measurev1.TopNRequest {
+ return b.req
+}
+
+func Eq(key string, val interface{}) *modelv1.Condition {
+ return &modelv1.Condition{
+ Name: key,
+ Op: modelv1.Condition_BINARY_OP_EQ,
+ Value: buildTagValue(val),
+ }
+}
+
+func Ne(key string, val interface{}) *modelv1.Condition {
+ return &modelv1.Condition{
+ Name: key,
+ Op: modelv1.Condition_BINARY_OP_NE,
+ Value: buildTagValue(val),
+ }
+}
+
+func Le(key string, val interface{}) *modelv1.Condition {
+ return &modelv1.Condition{
+ Name: key,
+ Op: modelv1.Condition_BINARY_OP_LE,
+ Value: buildTagValue(val),
+ }
+}
+
+func Lt(key string, val interface{}) *modelv1.Condition {
+ return &modelv1.Condition{
+ Name: key,
+ Op: modelv1.Condition_BINARY_OP_LT,
+ Value: buildTagValue(val),
+ }
+}
+
+func Ge(key string, val interface{}) *modelv1.Condition {
+ return &modelv1.Condition{
+ Name: key,
+ Op: modelv1.Condition_BINARY_OP_GE,
+ Value: buildTagValue(val),
+ }
+}
+
+func Gt(key string, val interface{}) *modelv1.Condition {
+ return &modelv1.Condition{
+ Name: key,
+ Op: modelv1.Condition_BINARY_OP_GT,
+ Value: buildTagValue(val),
+ }
+}
+
+func And(tagFamilyName string, conditions ...*modelv1.Condition) *modelv1.Criteria {
+ return &modelv1.Criteria{
+ TagFamilyName: tagFamilyName,
+ Conditions: conditions,
+ }
+}
diff --git a/pkg/query/aggregation/aggregation.go b/pkg/query/aggregation/aggregation.go
index 708db22..69d9320 100644
--- a/pkg/query/aggregation/aggregation.go
+++ b/pkg/query/aggregation/aggregation.go
@@ -14,9 +14,12 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package aggregation
import (
+ "math"
+
"github.com/pkg/errors"
modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
@@ -37,9 +40,13 @@ func NewInt64Func(af modelv1.AggregationFunction) (Int64Func, error) {
case modelv1.AggregationFunction_AGGREGATION_FUNCTION_COUNT:
return &countInt64Func{}, nil
case modelv1.AggregationFunction_AGGREGATION_FUNCTION_MAX:
- return &maxInt64Func{}, nil
+ return &maxInt64Func{
+ val: math.MinInt64,
+ }, nil
case modelv1.AggregationFunction_AGGREGATION_FUNCTION_MIN:
- return &minInt64Func{}, nil
+ return &minInt64Func{
+ val: math.MaxInt64,
+ }, nil
case modelv1.AggregationFunction_AGGREGATION_FUNCTION_SUM:
return &sumInt64Func{}, nil
}
diff --git a/pkg/query/aggregation/function.go b/pkg/query/aggregation/function.go
index 5144a81..59bdbc8 100644
--- a/pkg/query/aggregation/function.go
+++ b/pkg/query/aggregation/function.go
@@ -14,6 +14,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package aggregation
import "math"
diff --git a/pkg/query/logical/common.go b/pkg/query/logical/common.go
index a473a0a..b2e88e3 100644
--- a/pkg/query/logical/common.go
+++ b/pkg/query/logical/common.go
@@ -94,10 +94,10 @@ func projectItem(ec executor.ExecutionContext, item tsdb.Item, projectionFieldRe
// This method is used by the underlying tableScan and localIndexScan plans.
func executeForShard(series tsdb.SeriesList, timeRange timestamp.TimeRange,
builders ...seekerBuilder,
-) ([]tsdb.Iterator, error) {
- var itersInShard []tsdb.Iterator
+) ([]tsdb.ItemIterator, error) {
+ var itersInShard []tsdb.ItemIterator
for _, seriesFound := range series {
- itersInSeries, err := func() ([]tsdb.Iterator, error) {
+ itersInSeries, err := func() ([]tsdb.ItemIterator, error) {
sp, errInner := seriesFound.Span(timeRange)
defer func(sp tsdb.SeriesSpan) {
if sp != nil {
diff --git a/pkg/query/logical/expr.go b/pkg/query/logical/expr.go
index dccbd45..aa0ed45 100644
--- a/pkg/query/logical/expr.go
+++ b/pkg/query/logical/expr.go
@@ -36,6 +36,10 @@ var binaryOpFactory = map[modelv1.Condition_BinaryOp]func(l, r Expr) Expr{
modelv1.Condition_BINARY_OP_NOT_HAVING: NotHaving,
}
+func OpFactory(op modelv1.Condition_BinaryOp) func(l, r Expr) Expr {
+ return binaryOpFactory[op]
+}
+
var _ ResolvableExpr = (*TagRef)(nil)
// TagRef is the reference to the field
diff --git a/pkg/query/logical/iter.go b/pkg/query/logical/iter.go
index 739d237..6b960de 100644
--- a/pkg/query/logical/iter.go
+++ b/pkg/query/logical/iter.go
@@ -36,7 +36,7 @@ var _ heap.Interface = (*containerHeap)(nil)
type container struct {
c comparator
item tsdb.Item
- iter tsdb.Iterator
+ iter tsdb.ItemIterator
}
type containerHeap []*container
@@ -60,8 +60,8 @@ func (h *containerHeap) Pop() interface{} {
type itemIter struct {
// c is the comparator to sort items
c comparator
- // iters is the list of initial Iterator
- iters []tsdb.Iterator
+ // iters is the list of initial ItemIterator
+ iters []tsdb.ItemIterator
// deq is the deque of the container
// 1. When we push a new container, we can normally append it to the tail of the deq,
// and then sort the whole slice
@@ -70,7 +70,7 @@ type itemIter struct {
h *containerHeap
}
-func NewItemIter(iters []tsdb.Iterator, c comparator) ItemIterator {
+func NewItemIter(iters []tsdb.ItemIterator, c comparator) ItemIterator {
it := &itemIter{
c: c,
iters: iters,
@@ -92,7 +92,7 @@ func (it *itemIter) init() {
}
// pushIterator pushes the given iterator into the underlying deque.
-// Status will be immediately checked if the Iterator has a next value.
+// Status will be immediately checked if the ItemIterator has a next value.
// 1 - If not, it will be close at once and will not be added to the slice,
//
// which means inactive iterator does not exist in the deq.
@@ -100,13 +100,14 @@ func (it *itemIter) init() {
// 2 - If so, it will be wrapped into a container and push to the deq.
//
// Then we call SliceStable sort to sort the deq.
-func (it *itemIter) pushIterator(iter tsdb.Iterator) {
- if !iter.Next() {
+func (it *itemIter) pushIterator(iter tsdb.ItemIterator) {
+ item, hasNext := iter.Next()
+ if !hasNext {
_ = iter.Close()
return
}
heap.Push(it.h, &container{
- item: iter.Val(),
+ item: item,
iter: iter,
c: it.c,
})
diff --git a/pkg/query/logical/measure_plan_aggregation.go b/pkg/query/logical/measure_plan_aggregation.go
index e665156..d4f5b71 100644
--- a/pkg/query/logical/measure_plan_aggregation.go
+++ b/pkg/query/logical/measure_plan_aggregation.go
@@ -14,6 +14,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package logical
import (
diff --git a/pkg/query/logical/measure_plan_indexscan_local.go b/pkg/query/logical/measure_plan_indexscan_local.go
index 78a9c83..a9bb58b 100644
--- a/pkg/query/logical/measure_plan_indexscan_local.go
+++ b/pkg/query/logical/measure_plan_indexscan_local.go
@@ -51,9 +51,9 @@ type unresolvedMeasureIndexScan struct {
unresolvedOrderBy *UnresolvedOrderBy
}
-func (uis *unresolvedMeasureIndexScan) Analyze(s Schema) (Plan, error) {
+func ParseConditionMap(conditions []Expr, s Schema) (map[*databasev1.IndexRule][]Expr, error) {
localConditionMap := make(map[*databasev1.IndexRule][]Expr)
- for _, cond := range uis.conditions {
+ for _, cond := range conditions {
if resolvable, ok := cond.(ResolvableExpr); ok {
err := resolvable.Resolve(s)
if err != nil {
@@ -78,21 +78,30 @@ func (uis *unresolvedMeasureIndexScan) Analyze(s Schema) (Plan, error) {
}
}
+ return localConditionMap, nil
+}
+
+func (uis *unresolvedMeasureIndexScan) Analyze(s Schema) (Plan, error) {
+ conditionMap, err := ParseConditionMap(uis.conditions, s)
+ if err != nil {
+ return nil, err
+ }
+
var projTagsRefs [][]*TagRef
if len(uis.projectionTags) > 0 {
- var err error
- projTagsRefs, err = s.CreateTagRef(uis.projectionTags...)
- if err != nil {
- return nil, err
+ var innerErr error
+ projTagsRefs, innerErr = s.CreateTagRef(uis.projectionTags...)
+ if innerErr != nil {
+ return nil, innerErr
}
}
var projFieldRefs []*FieldRef
if len(uis.projectionFields) > 0 {
- var err error
- projFieldRefs, err = s.CreateFieldRef(uis.projectionFields...)
- if err != nil {
- return nil, err
+ var innerErr error
+ projFieldRefs, innerErr = s.CreateFieldRef(uis.projectionFields...)
+ if innerErr != nil {
+ return nil, innerErr
}
}
@@ -108,7 +117,7 @@ func (uis *unresolvedMeasureIndexScan) Analyze(s Schema) (Plan, error) {
projectionTagsRefs: projTagsRefs,
projectionFieldsRefs: projFieldRefs,
metadata: uis.metadata,
- conditionMap: localConditionMap,
+ conditionMap: conditionMap,
entity: uis.entity,
groupByEntity: uis.groupByEntity,
orderBy: orderBySubPlan,
@@ -134,7 +143,7 @@ func (i *localMeasureIndexScan) Execute(ec executor.MeasureExecutionContext) (ex
if err != nil {
return nil, err
}
- var iters []tsdb.Iterator
+ var iters []tsdb.ItemIterator
for _, shard := range shards {
itersInShard, innerErr := i.executeInShard(shard)
if innerErr != nil {
@@ -162,7 +171,7 @@ func (i *localMeasureIndexScan) Execute(ec executor.MeasureExecutionContext) (ex
return newIndexScanMIterator(it, transformContext), nil
}
-func (i *localMeasureIndexScan) executeInShard(shard tsdb.Shard) ([]tsdb.Iterator, error) {
+func (i *localMeasureIndexScan) executeInShard(shard tsdb.Shard) ([]tsdb.ItemIterator, error) {
seriesList, err := shard.Series().List(tsdb.NewPath(i.entity))
if err != nil {
return nil, err
@@ -195,7 +204,7 @@ func (i *localMeasureIndexScan) executeInShard(shard tsdb.Shard) ([]tsdb.Iterato
if i.conditionMap != nil && len(i.conditionMap) > 0 {
builders = append(builders, func(b tsdb.SeekerBuilder) {
for idxRule, exprs := range i.conditionMap {
- b.Filter(idxRule, exprToCondition(exprs))
+ b.Filter(idxRule, ExprToCondition(exprs))
}
})
}
@@ -316,7 +325,7 @@ func (ism *indexScanMIterator) Close() error {
var _ executor.MIterator = (*seriesMIterator)(nil)
type seriesMIterator struct {
- inner []tsdb.Iterator
+ inner []tsdb.ItemIterator
context transformContext
index int
@@ -324,7 +333,7 @@ type seriesMIterator struct {
err error
}
-func newSeriesMIterator(inner []tsdb.Iterator, context transformContext) executor.MIterator {
+func newSeriesMIterator(inner []tsdb.ItemIterator, context transformContext) executor.MIterator {
return &seriesMIterator{
inner: inner,
context: context,
@@ -344,8 +353,12 @@ func (ism *seriesMIterator) Next() bool {
if ism.current != nil {
ism.current = ism.current[:0]
}
- for iter.Next() {
- dp, err := transform(iter.Val(), ism.context)
+ for {
+ val, hasNext := iter.Next()
+ if !hasNext {
+ break
+ }
+ dp, err := transform(val, ism.context)
if err != nil {
ism.err = err
return false
diff --git a/pkg/query/logical/stream_plan_indexscan_local.go b/pkg/query/logical/stream_plan_indexscan_local.go
index bec23df..462415b 100644
--- a/pkg/query/logical/stream_plan_indexscan_local.go
+++ b/pkg/query/logical/stream_plan_indexscan_local.go
@@ -52,7 +52,7 @@ func (i *localIndexScan) Execute(ec executor.StreamExecutionContext) ([]*streamv
if err != nil {
return nil, err
}
- var iters []tsdb.Iterator
+ var iters []tsdb.ItemIterator
for _, shard := range shards {
itersInShard, innerErr := i.executeInShard(shard)
if innerErr != nil {
@@ -91,7 +91,7 @@ func (i *localIndexScan) Execute(ec executor.StreamExecutionContext) ([]*streamv
return elems, nil
}
-func (i *localIndexScan) executeInShard(shard tsdb.Shard) ([]tsdb.Iterator, error) {
+func (i *localIndexScan) executeInShard(shard tsdb.Shard) ([]tsdb.ItemIterator, error) {
seriesList, err := shard.Series().List(tsdb.NewPath(i.entity))
if err != nil {
return nil, err
@@ -116,7 +116,7 @@ func (i *localIndexScan) executeInShard(shard tsdb.Shard) ([]tsdb.Iterator, erro
if i.conditionMap != nil && len(i.conditionMap) > 0 {
builders = append(builders, func(b tsdb.SeekerBuilder) {
for idxRule, exprs := range i.conditionMap {
- b.Filter(idxRule, exprToCondition(exprs))
+ b.Filter(idxRule, ExprToCondition(exprs))
}
})
}
@@ -174,7 +174,7 @@ func (i *localIndexScan) Equal(plan Plan) bool {
cmp.Equal(i.orderBy, other.orderBy)
}
-func exprToCondition(exprs []Expr) tsdb.Condition {
+func ExprToCondition(exprs []Expr) tsdb.Condition {
cond := make(map[string][]index.ConditionValue)
for _, expr := range exprs {
bExpr := expr.(*binaryExpr)
diff --git a/pkg/schema/metadata.go b/pkg/schema/metadata.go
index 7e92376..2d0111f 100644
--- a/pkg/schema/metadata.go
+++ b/pkg/schema/metadata.go
@@ -70,8 +70,11 @@ type ResourceSchema interface {
}
type ResourceSpec struct {
- Schema ResourceSchema
+ Schema ResourceSchema
+ // IndexRules are index rules bound to the Schema
IndexRules []*databasev1.IndexRule
+ // Aggregations are topN aggregation bound to the Schema, e.g. TopNAggregation
+ Aggregations []*databasev1.TopNAggregation
}
type Resource interface {
@@ -185,8 +188,8 @@ func (sr *schemaRepo) Watcher() {
func (sr *schemaRepo) StoreGroup(groupMeta *commonv1.Metadata) (*group, error) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
groupSchema, err := sr.metadata.GroupRegistry().GetGroup(ctx, groupMeta.GetName())
- cancel()
if err != nil {
return nil, err
}
@@ -431,14 +434,27 @@ func (g *group) StoreResource(resourceSchema ResourceSchema) (Resource, error) {
}
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
- idxRules, errIndexRules := g.metadata.IndexRules(ctx, resourceSchema.GetMetadata())
- cancel()
- if errIndexRules != nil {
- return nil, errIndexRules
+ defer cancel()
+ idxRules, err := g.metadata.IndexRules(ctx, resourceSchema.GetMetadata())
+ if err != nil {
+ return nil, err
}
+
+ var topNAggrs []*databasev1.TopNAggregation
+ if _, ok := resourceSchema.(*databasev1.Measure); ok {
+ ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
+ var innerErr error
+ topNAggrs, innerErr = g.metadata.TopNAggregations(ctx, resourceSchema.GetMetadata())
+ cancel()
+ if innerErr != nil {
+ return nil, innerErr
+ }
+ }
+
sm, errTS := g.resourceSupplier.OpenResource(g.groupSchema.GetResourceOpts().ShardNum, g, ResourceSpec{
- Schema: resourceSchema,
- IndexRules: idxRules,
+ Schema: resourceSchema,
+ IndexRules: idxRules,
+ Aggregations: topNAggrs,
})
if errTS != nil {
return nil, errTS
diff --git a/pkg/test/flow/slice.go b/pkg/test/flow/slice.go
new file mode 100644
index 0000000..1792d2e
--- /dev/null
+++ b/pkg/test/flow/slice.go
@@ -0,0 +1,85 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package flow
+
+import (
+ "context"
+ "reflect"
+
+ "github.com/pkg/errors"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+)
+
+var _ flow.Source = (*sourceSlice)(nil)
+
+type sourceSlice struct {
+ flow.ComponentState
+ slice interface{}
+ out chan flow.StreamRecord
+}
+
+func (s *sourceSlice) Out() <-chan flow.StreamRecord {
+ return s.out
+}
+
+func (s *sourceSlice) Setup(ctx context.Context) error {
+ // ensure slice param is a Slice type
+ dataType := reflect.TypeOf(s.slice)
+ if dataType.Kind() != reflect.Slice {
+ return errors.New("sourceSlice must have a slice")
+ }
+ sliceVal := reflect.ValueOf(s.slice)
+
+ go s.run(ctx, sliceVal)
+ return nil
+}
+
+func (s *sourceSlice) run(ctx context.Context, sliceVal reflect.Value) {
+ ctx, cancel := context.WithCancel(ctx)
+ defer func() {
+ cancel()
+ close(s.out)
+ }()
+
+ for i := 0; i < sliceVal.Len(); i++ {
+ val := sliceVal.Index(i)
+ select {
+ case s.out <- flow.TryExactTimestamp(val.Interface()):
+ case <-ctx.Done():
+ return
+ }
+ }
+}
+
+func (s *sourceSlice) Teardown(ctx context.Context) error {
+ s.Wait()
+ return nil
+}
+
+func (s *sourceSlice) Exec(downstream flow.Inlet) {
+ s.Add(1)
+ go flow.Transmit(&s.ComponentState, downstream, s)
+}
+
+func NewSlice(slice interface{}) flow.Source {
+ return &sourceSlice{
+ slice: slice,
+ out: make(chan flow.StreamRecord),
+ }
+}
diff --git a/pkg/test/flow/slice_test.go b/pkg/test/flow/slice_test.go
new file mode 100644
index 0000000..cc40294
--- /dev/null
+++ b/pkg/test/flow/slice_test.go
@@ -0,0 +1,63 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package flow_test
+
+import (
+ "context"
+ "strings"
+ "testing"
+
+ "github.com/golang/mock/gomock"
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/pkg/flow"
+ flowTest "github.com/apache/skywalking-banyandb/pkg/test/flow"
+)
+
+const (
+ ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+)
+
+func TestSource_slice(t *testing.T) {
+ ctrl := gomock.NewController(t)
+ defer ctrl.Finish()
+
+ inlet := flow.NewMockInlet(ctrl)
+
+ assert := require.New(t)
+ src := flowTest.NewSlice(strings.Split(ALPHABET, ""))
+ assert.NoError(src.Setup(context.TODO()))
+
+ in := make(chan flow.StreamRecord)
+ inlet.
+ EXPECT().
+ In().Times(len(ALPHABET) + 1).
+ Return(in)
+
+ src.Exec(inlet)
+
+ var result strings.Builder
+ for item := range in {
+ assert.IsType(flow.StreamRecord{}, item)
+ result.WriteString(item.Data().(string))
+ }
+
+ assert.Equal(ALPHABET, result.String())
+
+ assert.NoError(src.Teardown(context.TODO()))
+}
diff --git a/pkg/test/measure/etcd.go b/pkg/test/measure/etcd.go
index ee29262..5ed5a74 100644
--- a/pkg/test/measure/etcd.go
+++ b/pkg/test/measure/etcd.go
@@ -43,33 +43,38 @@ const (
measureDir = "testdata/measures"
indexRuleDir = "testdata/index_rules"
indexRuleBindingDir = "testdata/index_rule_bindings"
+ topNAggregationDir = "testdata/topn_aggregations"
)
//go:embed testdata/*
var store embed.FS
func PreloadSchema(e schema.Registry) error {
- if err := loadSchema(groupDir, &commonv1.Group{}, func(group proto.Message) error {
- return e.CreateGroup(context.TODO(), group.(*commonv1.Group))
+ if err := loadSchema(groupDir, &commonv1.Group{}, func(group *commonv1.Group) error {
+ return e.CreateGroup(context.TODO(), group)
}); err != nil {
return errors.WithStack(err)
}
- if err := loadSchema(measureDir, &databasev1.Measure{}, func(measure proto.Message) error {
- return e.CreateMeasure(context.TODO(), measure.(*databasev1.Measure))
+ if err := loadSchema(measureDir, &databasev1.Measure{}, func(measure *databasev1.Measure) error {
+ return e.CreateMeasure(context.TODO(), measure)
}); err != nil {
return errors.WithStack(err)
}
- if err := loadSchema(indexRuleDir, &databasev1.IndexRule{}, func(indexRule proto.Message) error {
- return e.CreateIndexRule(context.TODO(), indexRule.(*databasev1.IndexRule))
+ if err := loadSchema(indexRuleDir, &databasev1.IndexRule{}, func(indexRule *databasev1.IndexRule) error {
+ return e.CreateIndexRule(context.TODO(), indexRule)
}); err != nil {
return errors.WithStack(err)
}
- if err := loadSchema(indexRuleBindingDir, &databasev1.IndexRuleBinding{}, func(indexRuleBinding proto.Message) error {
- return e.CreateIndexRuleBinding(context.TODO(), indexRuleBinding.(*databasev1.IndexRuleBinding))
+ if err := loadSchema(indexRuleBindingDir, &databasev1.IndexRuleBinding{}, func(indexRuleBinding *databasev1.IndexRuleBinding) error {
+ return e.CreateIndexRuleBinding(context.TODO(), indexRuleBinding)
+ }); err != nil {
+ return errors.WithStack(err)
+ }
+ if err := loadSchema(topNAggregationDir, &databasev1.TopNAggregation{}, func(topN *databasev1.TopNAggregation) error {
+ return e.CreateTopNAggregation(context.TODO(), topN)
}); err != nil {
return errors.WithStack(err)
}
-
return nil
}
@@ -77,7 +82,7 @@ func RandomTempDir() string {
return path.Join(os.TempDir(), fmt.Sprintf("banyandb-embed-etcd-%s", uuid.New().String()))
}
-func loadSchema(dir string, resource proto.Message, loadFn func(resource proto.Message) error) error {
+func loadSchema[T proto.Message](dir string, resource T, loadFn func(resource T) error) error {
entries, err := store.ReadDir(dir)
if err != nil {
return err
@@ -109,20 +114,20 @@ func RegisterForNew(addr string, metricNum int) error {
ctx := context.Background()
- if err := loadSchema(groupDir, &commonv1.Group{}, func(group proto.Message) error {
+ if err := loadSchema(groupDir, &commonv1.Group{}, func(group *commonv1.Group) error {
return grpchelper.Request(ctx, rpcTimeout, func(rpcCtx context.Context) (err error) {
_, err = databasev1.NewGroupRegistryServiceClient(conn).
Create(rpcCtx, &databasev1.GroupRegistryServiceCreateRequest{
- Group: group.(*commonv1.Group),
+ Group: group,
})
return err
})
}); err != nil {
return errors.WithStack(err)
}
- if err := loadSchema(measureDir, &databasev1.Measure{}, func(measure proto.Message) error {
+ if err := loadSchema(measureDir, &databasev1.Measure{}, func(measure *databasev1.Measure) error {
var err error
- name := measure.(*databasev1.Measure).Metadata.Name
+ name := measure.GetMetadata().GetName()
num := metricNum
if name != "service_cpm_minute" {
num = 1
@@ -131,7 +136,7 @@ func RegisterForNew(addr string, metricNum int) error {
err = multierr.Append(err, grpchelper.Request(ctx, rpcTimeout, func(rpcCtx context.Context) (err error) {
m := proto.Clone(measure).(*databasev1.Measure)
if i > 0 {
- m.Metadata.Name = m.Metadata.Name + "_" + strconv.Itoa(i)
+ m.Metadata.Name = m.GetMetadata().GetName() + "_" + strconv.Itoa(i)
}
_, err = databasev1.NewMeasureRegistryServiceClient(conn).
Create(rpcCtx, &databasev1.MeasureRegistryServiceCreateRequest{
@@ -145,22 +150,22 @@ func RegisterForNew(addr string, metricNum int) error {
return errors.WithStack(err)
}
- if err := loadSchema(indexRuleDir, &databasev1.IndexRule{}, func(indexRule proto.Message) error {
+ if err := loadSchema(indexRuleDir, &databasev1.IndexRule{}, func(indexRule *databasev1.IndexRule) error {
return grpchelper.Request(ctx, rpcTimeout, func(rpcCtx context.Context) (err error) {
_, err = databasev1.NewIndexRuleRegistryServiceClient(conn).
Create(rpcCtx, &databasev1.IndexRuleRegistryServiceCreateRequest{
- IndexRule: indexRule.(*databasev1.IndexRule),
+ IndexRule: indexRule,
})
return err
})
}); err != nil {
return errors.WithStack(err)
}
- if err := loadSchema(indexRuleBindingDir, &databasev1.IndexRuleBinding{}, func(indexRuleBinding proto.Message) error {
+ if err := loadSchema(indexRuleBindingDir, &databasev1.IndexRuleBinding{}, func(indexRuleBinding *databasev1.IndexRuleBinding) error {
return grpchelper.Request(ctx, rpcTimeout, func(rpcCtx context.Context) (err error) {
_, err = databasev1.NewIndexRuleBindingRegistryServiceClient(conn).
Create(rpcCtx, &databasev1.IndexRuleBindingRegistryServiceCreateRequest{
- IndexRuleBinding: indexRuleBinding.(*databasev1.IndexRuleBinding),
+ IndexRuleBinding: indexRuleBinding,
})
return err
})
diff --git a/pkg/test/measure/testdata/topn_aggregations/service_cpm_minute_nogroup_top100.json b/pkg/test/measure/testdata/topn_aggregations/service_cpm_minute_nogroup_top100.json
new file mode 100644
index 0000000..91036f4
--- /dev/null
+++ b/pkg/test/measure/testdata/topn_aggregations/service_cpm_minute_nogroup_top100.json
@@ -0,0 +1,13 @@
+{
+ "metadata": {
+ "name": "service_cpm_minute_no_group_by_top100",
+ "group": "sw_metric"
+ },
+ "source_measure": {
+ "name": "service_cpm_minute",
+ "group": "sw_metric"
+ },
+ "field_name": "value",
+ "field_value_sort": 1,
+ "counters_number": 1000
+}
\ No newline at end of file
diff --git a/pkg/test/measure/testdata/topn_aggregations/service_cpm_minute_top100.json b/pkg/test/measure/testdata/topn_aggregations/service_cpm_minute_top100.json
new file mode 100644
index 0000000..5758ac9
--- /dev/null
+++ b/pkg/test/measure/testdata/topn_aggregations/service_cpm_minute_top100.json
@@ -0,0 +1,16 @@
+{
+ "metadata": {
+ "name": "service_cpm_minute_top100",
+ "group": "sw_metric"
+ },
+ "source_measure": {
+ "name": "service_cpm_minute",
+ "group": "sw_metric"
+ },
+ "field_name": "value",
+ "field_value_sort": 1,
+ "group_by_tag_names": [
+ "entity_id"
+ ],
+ "counters_number": 1000
+}
\ No newline at end of file
diff --git a/pkg/timestamp/duration.go b/pkg/timestamp/duration.go
index 8b07aa2..d345d9c 100644
--- a/pkg/timestamp/duration.go
+++ b/pkg/timestamp/duration.go
@@ -14,6 +14,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package timestamp
import (
diff --git a/pkg/timestamp/nano.go b/pkg/timestamp/nano.go
index c83decb..4f4cdab 100644
--- a/pkg/timestamp/nano.go
+++ b/pkg/timestamp/nano.go
@@ -14,6 +14,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package timestamp
import (
diff --git a/pkg/timestamp/range.go b/pkg/timestamp/range.go
index 44f0bc0..3c5b77b 100644
--- a/pkg/timestamp/range.go
+++ b/pkg/timestamp/range.go
@@ -14,6 +14,7 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
+
package timestamp
import (