You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/04/14 02:22:52 UTC
[GitHub] [pulsar] shibd opened a new issue, #15165: Pulsar SQL can't query the latest data
shibd opened a new issue, #15165:
URL: https://github.com/apache/pulsar/issues/15165
**Describe the bug**
Pulsar SQL can't query the latest data.
There are 3 pieces of data in the topic.
```
➜ pulsar git:(master) pulsar-admin topics peek-messages -n 10 -s test-sub persistent://public/default/test_avro3
2022-04-14T10:17:14,839+0800 [AsyncHttpClient-7-1] WARN org.apache.pulsar.client.admin.internal.TopicsImpl - Exception 'Message not found' occurred while trying to peek Messages.
Batch Message ID: 16:0:0
Publish time: 1649902603193
Event time: 0
Properties:
X-Pulsar-batch-size 28
X-Pulsar-num-batch-message 1
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 02 0a 62 61 6f 7a 69 02 02 31 |..baozi..1 |
+--------+-------------------------------------------------+----------------+
-------------------------------------------------------------------------
Batch Message ID: 16:1:0
Publish time: 1649902613175
Event time: 0
Properties:
X-Pulsar-batch-size 28
X-Pulsar-num-batch-message 1
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 02 0a 62 61 6f 7a 69 02 02 32 |..baozi..2 |
+--------+-------------------------------------------------+----------------+
-------------------------------------------------------------------------
Batch Message ID: 16:2:0
Publish time: 1649902624326
Event time: 0
Properties:
X-Pulsar-batch-size 28
X-Pulsar-num-batch-message 1
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 02 0a 62 61 6f 7a 69 02 02 33 |..baozi..3 |
+--------+-------------------------------------------------+----------------+
```
Using Pulsar SQL query always missing the lastest data.
```
presto> select * from pulsar."public/default".test_avro3;
name | value | __partition__ | __event_time__ | __publish_time__ | __message_id__ | __sequence_id__ | __producer_name__ | __key__ | __properties__
-------+-------+---------------+----------------+-------------------------+----------------+-----------------+-------------------+---------+----------------
baozi | 2 | -1 | NULL | 2022-04-14 10:16:53.175 | (16,1,0) | 0 | standalone-0-5 | NULL | {}
baozi | 1 | -1 | NULL | 2022-04-14 10:16:43.193 | (16,0,0) | 0 | standalone-0-4 | NULL | {}
```
**To Reproduce**
1. Produce 3 pieces of data to the topic.
2. Use Pulsq SQL query.
4. only see 2 row.
**Expected behavior**
Can query all data.
**Desktop (please complete the following information):**
- OS: MacOS
- Version: master brnach
- Run on standalone mode
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zymap commented on issue #15165: Pulsar SQL can't query the latest data
Posted by GitBox <gi...@apache.org>.
zymap commented on issue #15165:
URL: https://github.com/apache/pulsar/issues/15165#issuecomment-1098669461
https://pulsar.apache.org/docs/en/sql-deployment-configurations/#configure-presto-pulsar-connector
If you want to get the last message in a topic, set the following configurations:
For the broker configuration, set bookkeeperExplicitLacIntervalInMills > 0 in broker.conf or standalone.conf.
For the Presto configuration, set pulsar.bookkeeper-explicit-interval > 0 and pulsar.bookkeeper-use-v2-protocol=false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd commented on issue #15165: Pulsar SQL can't query the latest data
Posted by GitBox <gi...@apache.org>.
shibd commented on issue #15165:
URL: https://github.com/apache/pulsar/issues/15165#issuecomment-1098674234
> https://pulsar.apache.org/docs/en/sql-deployment-configurations/#configure-presto-pulsar-connector
If you want to get the last message in a topic, set the following configurations:
For the broker configuration, set bookkeeperExplicitLacIntervalInMills > 0 in broker.conf or standalone.conf.
For the Presto configuration, set pulsar.bookkeeper-explicit-interval > 0 and pulsar.bookkeeper-use-v2-protocol=false.
Thanks, That was useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] shibd closed issue #15165: Pulsar SQL can't query the latest data
Posted by GitBox <gi...@apache.org>.
shibd closed issue #15165: Pulsar SQL can't query the latest data
URL: https://github.com/apache/pulsar/issues/15165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org