You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "crossoverJie (via GitHub)" <gi...@apache.org> on 2023/08/02 10:47:34 UTC
[GitHub] [pulsar] crossoverJie opened a new issue, #20919: [Bug] When using publish_time as a query condition in Pulsar SQL, the query result would lose one data record.
crossoverJie opened a new issue, #20919:
URL: https://github.com/apache/pulsar/issues/20919
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
master
### Minimal reproduce step
Raw data:
```
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:33:15.274 | (449,0,0) | 0 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 31 | -1 | NULL | 2023-08-02 07:33:25.305 | (449,1,0) | 1 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
```
SQL:
```sql
select * from pulsar."public/default"."topic10" WHERE "__publish_time__" > TIMESTAMP '2023-08-02 07:33:35' AND "__publish_time__" < TIMESTAMP '2023-08-02 07:46:51';
```
Result:
```
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
(8 rows)
```
### What did you expect to see?
```
68 65 6c 6c 6f 32 | -1 | NULL | 2023-08-02 07:33:35.314 | (449,2,0) | 2 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 33 | -1 | NULL | 2023-08-02 07:33:45.324 | (449,3,0) | 3 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 34 | -1 | NULL | 2023-08-02 07:33:55.336 | (449,4,0) | 4 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 35 | -1 | NULL | 2023-08-02 07:34:05.345 | (449,5,0) | 5 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 36 | -1 | NULL | 2023-08-02 07:34:15.355 | (449,6,0) | 6 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 37 | -1 | NULL | 2023-08-02 07:34:25.395 | (449,7,0) | 7 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 38 | -1 | NULL | 2023-08-02 07:34:35.404 | (449,8,0) | 8 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 39 | -1 | NULL | 2023-08-02 07:34:45.412 | (449,9,0) | 9 | standalone-8-1 | NULL | {}
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
(9 rows)
```
### What did you see instead?
The missing data record:
```
68 65 6c 6c 6f 30 | -1 | NULL | 2023-08-02 07:46:50.961 | (449,10,0) | 0 | standalone-8-2 | NULL | {}
```
### Anything else?
https://github.com/apache/pulsar/blob/782e91fe327efe2c9c9107d6c679c2837d43935b/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L420-L423
I discovered that the query range reduced 1 at the last, after removing it, the running looks well.
Why `-1`, any special considerations?
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using publish_time as a query condition in Pulsar SQL, the query result would lose one data record. [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #20919:
URL: https://github.com/apache/pulsar/issues/20919#issuecomment-6377102263
Thanks @crossoverJie for the clear reproduction. Pulsar SQL was removed from the main distribution in 3.2.0 (#21795) and now lives in apache/pulsar-sql. The `__publish_time__` pushdown in PulsarSplitManager there still subtracts one from the number of entries in a closed position range, which fits the record you lost. Moving this issue to apache/pulsar-sql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #20919: [Bug] When using publish_time as a query condition in Pulsar SQL, the query result would lose one data record.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #20919:
URL: https://github.com/apache/pulsar/issues/20919#issuecomment-1703639272
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org