You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "saurabhd336 (via GitHub)" <gi...@apache.org> on 2023/07/24 10:59:58 UTC
[GitHub] [pinot] saurabhd336 opened a new pull request, #11158: Increment nextDocId even if geo indexing fails
saurabhd336 opened a new pull request, #11158:
URL: https://github.com/apache/pinot/pull/11158
If geo indexing fails for any row in a consuming segment, wrong docIds are put into the index for all rows that follow. This fixes the behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11158: Increment nextDocId even if geo indexing fails
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11158:
URL: https://github.com/apache/pinot/pull/11158#issuecomment-1647851587
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11158?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11158](https://app.codecov.io/gh/apache/pinot/pull/11158?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (874aabb) into [master](https://app.codecov.io/gh/apache/pinot/commit/723b764bc91275c0b8361d3f9135f151b6404c39?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (723b764) will **decrease** coverage by `0.01%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11158 +/- ##
==========================================
- Coverage 0.11% 0.11% -0.01%
==========================================
Files 2205 2208 +3
Lines 118328 118584 +256
Branches 17908 17944 +36
==========================================
Hits 137 137
- Misses 118171 118427 +256
Partials 20 20
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1temurin11 | `0.00% <0.00%> (ø)` | |
| integration1temurin17 | `0.00% <0.00%> (ø)` | |
| integration1temurin20 | `0.00% <0.00%> (ø)` | |
| integration2temurin11 | `?` | |
| integration2temurin17 | `?` | |
| integration2temurin20 | `0.00% <0.00%> (ø)` | |
| unittests1temurin11 | `0.00% <0.00%> (?)` | |
| unittests1temurin17 | `0.00% <0.00%> (ø)` | |
| unittests1temurin20 | `0.00% <0.00%> (ø)` | |
| unittests2temurin11 | `0.11% <0.00%> (-0.01%)` | :arrow_down: |
| unittests2temurin17 | `0.11% <0.00%> (-0.01%)` | :arrow_down: |
| unittests2temurin20 | `0.11% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/pinot/pull/11158?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...local/realtime/impl/geospatial/MutableH3Index.java](https://app.codecov.io/gh/apache/pinot/pull/11158?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2dlb3NwYXRpYWwvTXV0YWJsZUgzSW5kZXguamF2YQ==) | `0.00% <0.00%> (ø)` | |
... and [26 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11158/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] saurabhd336 commented on pull request #11158: Increment nextDocId even if geo indexing fails
Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on PR #11158:
URL: https://github.com/apache/pinot/pull/11158#issuecomment-1647784240
Issue reproducible with
Schema
```
{
"schemaName": "geoIndex",
"dimensionFieldSpecs": [
{
"name": "geoIndexCol",
"dataType": "BYTES"
},
{
"name": "lon",
"dataType": "STRING"
},
{
"name": "lat",
"dataType": "STRING"
}
],
"dateTimeFieldSpecs": [{
"name": "timestampInEpoch",
"dataType": "LONG",
"format" : "1:MILLISECONDS:EPOCH",
"granularity": "1:MILLISECONDS"
}]
}
```
TableConfig
```
{
"tableName": "geoTest",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "timestampInEpoch",
"timeType": "MILLISECONDS",
"schemaName": "geoIndex",
"replicasPerPartition": "1"
},
"fieldConfigList": [{
"name": "geoIndexCol",
"encodingType": "RAW",
"indexType": "H3",
"indexTypes": [
"H3"
],
"properties": {
"resolutions": "5"
},
"tierOverwrites": null
}],
"tenants": {},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.topic.name": "geo-test-topic",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "localhost:19092",
"realtime.segment.flush.threshold.rows": "20",
"realtime.segment.flush.threshold.time": "10m",
"realtime.segment.flush.threshold.segment.size": "50M",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
},
"ingestionConfig": {
"transformConfigs": [{
"columnName": "geoIndexCol",
"transformFunction": "toSphericalGeography(stPoint(lon,lat))"
}]
},
"metadata": {
"customConfigs": {}
}
}
```
data.json
```
{"lat":"12.933540687183736","lon":"77.62542483501983","timestampInEpoch":1690192996000}
{"lat":null, "lon":"77.62542483501983","timestampInEpoch":1690192996000}
{"lat":null, "lon":"77.62542483501983","timestampInEpoch":1690192996000}
{"lat":"12.933540687183736","lon":"77.62542483501983","timestampInEpoch":1690192996000}
```
Query
```
SELECT * FROM geoTest where ST_DISTANCE(locationSTPoint, ST_Point(77.62357943770779, 12.933540687183736, 1)) < 500
```
Fails with exception without this fix since the mutable index has incorrect docIds in the h3Id -> bitmap mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11158: Increment nextDocId even if geo indexing fails
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11158:
URL: https://github.com/apache/pinot/pull/11158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org