You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rohityadav1993 (via GitHub)" <gi...@apache.org> on 2024/01/29 05:40:15 UTC
[I] Backfill for partial upsert tables. [pinot]
rohityadav1993 opened a new issue, #12331:
URL: https://github.com/apache/pinot/issues/12331
There are ways to upload pre-generated segments to Pinot tables. This provides a mechanism for backfills for realtime and offline tables (bootstrapping a new table or correcting data). If upsert metadata is properly provided for full upsert tables, then pre-partitioned segments by the primary key can also be uploaded with correct upsert behaviour.
The above approach falls short when uploading segments of partial upsert table as there is an insert time transformation on the row by merging it with previously persisted row for a primary key.
Instead of uploading pre-generated segments to partial upsert table, one can also not put the backfill data in the stream for realtime ingestion to take care of insert time transformation (merging) as during backfills one would want to remove all the rows of a primary key persisted and then have a fresh ingestion for the primary key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Backfill for partial upsert tables - need suggestions [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12331:
URL: https://github.com/apache/pinot/issues/12331#issuecomment-5690115875
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Backfill for partial upsert tables - need suggestions [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #12331: Backfill for partial upsert tables - need suggestions
URL: https://github.com/apache/pinot/issues/12331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Backfill for partial upsert tables - need suggestions [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12331:
URL: https://github.com/apache/pinot/issues/12331#issuecomment-5274340818
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org