You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2024/07/13 13:17:17 UTC
[PR] Support duplicated data for Stream [skywalking-banyandb]
hanahmily opened a new pull request, #485:
URL: https://github.com/apache/skywalking-banyandb/pull/485
### Fix a bug that the Stream module didn't support duplicated in index-based filtering and sorting
In the index, the document id holds the timestamp before these changes. This can lead to several incorrect reads when using the index to filter or sort the data, especially when there are multiple duplicated timestamps in a single time series.
In this pull request, I replaced the timestamp with the elementID, which is unique in a time series. The sorted elementID is now represented by the hashed uint64 value instead of the original string. The file system version is also updated due to this change.
### Fix missing data when a tag filter is applied
When sorting data by an index, if the final result size exceeds the offset+limit and a tag filter is applied, some elements may be lost in the query result.
In this pull request, the `limit` and `tag filter` executor can drain the parent source until no more data is returned or until the offset+limit is reached. This should help resolve the issue of missing data.
A new `distributed limit`, which is the same as the previous `limit`, is applied to the distributed executor. This is because the liaison couldn't drain data nodes several times; it should interact with each data node only once in a query request.
### Reverted Index Enhancement
The "Write" operation has been removed from the reverted index. Now, we can use the `Batch` operation to write index data.
The timestamp has been moved from the document id to a field with a date-time type, and its value is stored in the index.
@ButterBright Please check the query process to verify the change is expected.
- [x] Add a unit test to verify that the fix works.
- [x] Explain briefly why the bug exists and how to fix it.
- [x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Fixes apache/skywalking#12371.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support duplicated data for Stream [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on code in PR #485:
URL: https://github.com/apache/skywalking-banyandb/pull/485#discussion_r1677135652
##########
pkg/index/inverted/inverted.go:
##########
@@ -174,7 +183,7 @@ func (s *store) Iterator(fieldKey index.FieldKey, termRange index.RangeOpts, ord
if err != nil {
return nil, err
}
- fk := fieldKey.MarshalIndexRule()
+ fk := fieldKey.Marshal()
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support duplicated data for Stream [skywalking-banyandb]
Posted by "ButterBright (via GitHub)" <gi...@apache.org>.
ButterBright commented on code in PR #485:
URL: https://github.com/apache/skywalking-banyandb/pull/485#discussion_r1677015426
##########
pkg/index/inverted/inverted.go:
##########
@@ -174,7 +183,7 @@ func (s *store) Iterator(fieldKey index.FieldKey, termRange index.RangeOpts, ord
if err != nil {
return nil, err
}
- fk := fieldKey.MarshalIndexRule()
+ fk := fieldKey.Marshal()
Review Comment:
Please remove the line bluge.blugeMatchAllquery() from this function, as it's redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support duplicated data for Stream [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #485:
URL: https://github.com/apache/skywalking-banyandb/pull/485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org