You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/02/02 19:50:47 UTC
[GitHub] [incubator-pinot] icefury71 opened a new issue #6523: Support for configurable time column in Upserts
icefury71 opened a new issue #6523:
URL: https://github.com/apache/incubator-pinot/issues/6523
Currently the Pinot upserts functionality is using the primary time column to resolve the latest version of a record/document. In some cases, its desirable to make this configurable so that we can choose a custom resolution column.
CC @yupeng9 @Jackie-Jiang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] keweishang commented on issue #6523: Support for configurable time column in Upserts
Posted by GitBox <gi...@apache.org>.
keweishang commented on issue #6523:
URL: https://github.com/apache/incubator-pinot/issues/6523#issuecomment-872455648
We at Bolt also need this feature. We use Kafka Streams to join different Kafka topics. Kafka Streams' JOIN produces many Kafka records with the same event-time column value. Pinot randomly picks one record as the latest one when the event-time value is the same and it doesn't guarantee that the record of the latest Kafka offset is chosen.
It would be great to use other columns (e.g. an int/long column or Kafka Offset) than the event-time column to decide which is the last version when the primary key is the same.
We're evaluating Pinot at Bolt and this is a blocking issue for us at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on issue #6523: Support for configurable time column in Upserts
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on issue #6523:
URL: https://github.com/apache/incubator-pinot/issues/6523#issuecomment-873223277
I don't think it's hard. We need to add a configuration, and have the upsert metadata manager use it instead of the time column for determining the latest. I can take this when I have some cycyles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] keweishang edited a comment on issue #6523: Support for configurable time column in Upserts
Posted by GitBox <gi...@apache.org>.
keweishang edited a comment on issue #6523:
URL: https://github.com/apache/incubator-pinot/issues/6523#issuecomment-872455648
We at Bolt also need this feature. We use Kafka Streams to join different Kafka topics. Kafka Streams' JOIN produces many Kafka records with the same event-time column value. Pinot randomly picks one record as the latest one when the event-time value is the same and it doesn't guarantee that the record of the latest Kafka offset is chosen.
It would be great to use other columns (e.g. an int/long column or Kafka Offset) than the event-time column to decide which is the last version when the primary key is the same.
We're evaluating Pinot at Bolt and this is a blocking issue for us at the moment. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on issue #6523: Support for configurable time column in Upserts
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #6523:
URL: https://github.com/apache/incubator-pinot/issues/6523#issuecomment-873201972
Tagging @yupeng9 to see if this could be done easily?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org