You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/06 10:48:51 UTC
[GitHub] [pulsar] Hugues514 created a discussion: Pulsar SQL should support readCompact on topics
GitHub user Hugues514 created a discussion: Pulsar SQL should support readCompact on topics
So far, when a Pulsar topic is compacted, the Pulsar SQL query will read all the records, not considering the compacted option
It could be interesting to be able to query such a topic using a readCompact option so that the reader will skip all the 'compacted records'.
GitHub link: https://github.com/apache/pulsar/discussions/18764
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] jiazhai added a comment to the discussion: Pulsar SQL should support readCompact on topics
Posted by GitBox <gi...@apache.org>.
GitHub user jiazhai added a comment to the discussion: Pulsar SQL should support readCompact on topics
We need a special handle in `PulsarSplitManager.getSplitsForTopic` to split based also on `compactionHorizon`(contained in `CompactedTopicImpl`); so that for data before compactionHorizon, we could read from compacted topic, while data after compactionHorizon, read from non-compacted topic.
GitHub link: https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321919
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] jiazhai added a comment to the discussion: Pulsar SQL should support readCompact on topics
Posted by GitBox <gi...@apache.org>.
GitHub user jiazhai added a comment to the discussion: Pulsar SQL should support readCompact on topics
more info:
https://github.com/apache/pulsar/wiki/PIP-19:-Pulsar-SQL
https://github.com/apache/pulsar/wiki/PIP-14:-Topic-compaction
GitHub link: https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321920
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] Hugues514 edited a discussion: Wishing list: Pulsar SQL should support readCompact on topics
Posted by GitBox <gi...@apache.org>.
GitHub user Hugues514 edited a discussion: Wishing list: Pulsar SQL should support readCompact on topics
So far, when a Pulsar topic is compacted, the Pulsar SQL query will read all the records, not considering the compacted option
It could be interesting to be able to query such a topic using a readCompact option so that the reader will skip all the 'compacted records'.
GitHub link: https://github.com/apache/pulsar/discussions/18764
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] pointearth added a comment to the discussion: Pulsar SQL should support readCompact on topics
Posted by GitBox <gi...@apache.org>.
GitHub user pointearth added a comment to the discussion: Pulsar SQL should support readCompact on topics
Can you guys open this ticket and finish it or move it to a suitable appointment? appreciate!
[Add readCompact option #7461](https://github.com/apache/pulsar/pull/7461 )
GitHub link: https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321921
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Wishing list: Pulsar SQL should support readCompact on topics
Posted by GitBox <gi...@apache.org>.
GitHub user tisonkun added a comment to the discussion: Wishing list: Pulsar SQL should support readCompact on topics
Moved to general discussions since there's a wishing list item and no one works on it now.
GitHub link: https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321936
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org
[GitHub] [pulsar] longtengz added a comment to the discussion: Pulsar SQL should support readCompact on topics
Posted by GitBox <gi...@apache.org>.
GitHub user longtengz added a comment to the discussion: Pulsar SQL should support readCompact on topics
Does this also imply pulsar sql cannot process empty payload messages for the time being? Because I'm experiencing #13127
GitHub link: https://github.com/apache/pulsar/discussions/18764#discussioncomment-4321922
----
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@pulsar.apache.org