You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "0ZhangJc0 (via GitHub)" <gi...@apache.org> on 2023/06/13 05:31:32 UTC
[GitHub] [pinot] 0ZhangJc0 opened a new issue, #10903: Does Pinot support consuming multiple pieces of data from Kafka?
0ZhangJc0 opened a new issue, #10903:
URL: https://github.com/apache/pinot/issues/10903
In our usage scenario, a Kafka message is a compressed gz byte stream of multiple rows of data. Each time a message is read from Kafka, I need to decompress and split it into multiple rows of data, that is, multiple Generic Rows are included in one message Does Pinot support this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 0ZhangJc0 commented on issue #10903: Does Pinot support consuming multiple pieces of data from Kafka?
Posted by "0ZhangJc0 (via GitHub)" <gi...@apache.org>.
0ZhangJc0 commented on issue #10903:
URL: https://github.com/apache/pinot/issues/10903#issuecomment-1658145060
> @0ZhangJc0 If the entire payload is compressed by your application (and not by kafka), then we don't yet have an ability in Pinot connectors to de-compress the payload. It is not a typical use-case as kafka itself compresses record/record batches.
>
> I am curious to learn your use-case as the application is compressing the payload before publishing to kafka. can you share more details?
First of all, thanks for the quick reply. :)
In fact, we use nsq as the message middleware, and compress multi-line data to improve data transmission throughput
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #10903: Does Pinot support consuming multiple pieces of data from Kafka?
Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #10903:
URL: https://github.com/apache/pinot/issues/10903#issuecomment-1659298974
> In fact, we use nsq as the message middleware, and compress multi-line data to improve data transmission throughput
This is the first time I am hearing about nsq. Are you using kafka connector to consume from nsq? is nsq kafka client compatible?
In short, if nsq is kafka client compatible, then we can potentially explore and see if we can leverage the `$MULTIPLE_RECORD_KEYS` feature in the transformer pipeline to convert multi-line data into multi-records. But the kafka connector that exists today doesn't support it.
Alternatively, if nsq is not kafka client compatible, then the best option would be to write its own connector for pinot stream ingestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] 0ZhangJc0 commented on issue #10903: Does Pinot support consuming multiple pieces of data from Kafka?
Posted by "0ZhangJc0 (via GitHub)" <gi...@apache.org>.
0ZhangJc0 commented on issue #10903:
URL: https://github.com/apache/pinot/issues/10903#issuecomment-1650929760
> @0ZhangJc0 Pinot does not natively support 1:N consumption model (that is, where 1 record is consumed, but deserialized into multiple records). Is the entire record compressed or a specific field in the deserialized record that is compressed?
The entire record is compressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #10903: Does Pinot support consuming multiple pieces of data from Kafka?
Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #10903:
URL: https://github.com/apache/pinot/issues/10903#issuecomment-1597566698
@0ZhangJc0 Pinot does not natively support 1:N consumption model (that is, where 1 record is consumed, but deserialized into multiple records). Is the entire record compressed or a specific field in the deserialized record that is compressed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10903: Does Pinot support consuming multiple pieces of data from Kafka?
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10903:
URL: https://github.com/apache/pinot/issues/10903#issuecomment-1595465096
cc @snleee @npawar @navina
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] navina commented on issue #10903: Does Pinot support consuming multiple pieces of data from Kafka?
Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #10903:
URL: https://github.com/apache/pinot/issues/10903#issuecomment-1654206638
@0ZhangJc0 If the entire payload is compressed by your application (and not by kafka), then we don't yet have an ability in Pinot connectors to de-compress the payload. It is not a typical use-case as kafka itself compresses record/record batches.
I am curious to learn your use-case as the application is compressing the payload before publishing to kafka. can you share more details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org