You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Harnoor7 (via GitHub)" <gi...@apache.org> on 2024/12/13 20:20:50 UTC
[I] [Bug] Data Inconsistency issues in RealtimeToOffline Minion tasks [pinot]
Harnoor7 opened a new issue, #14659:
URL: https://github.com/apache/pinot/issues/14659
1. Currently for RTO tasks, watermark is updated in the executor after segments are uploaded. Hence, there can be possible scenarios where segments were uploaded to an offline table but RTO metadata watermark was not updated. AND RTO task generator does not validate if the segment has already been processed in the previous minion run.
2. If the time window (bucket size) of an RTO task is smaller than the segment time window, the same segment can be picked multiple times.
Due to above points, Offline table can have inconsistent data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Bug] Data Inconsistency in RealtimeToOffline Minion tasks [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14659:
URL: https://github.com/apache/pinot/issues/14659#issuecomment-4627182122
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Bug] Data Inconsistency in RealtimeToOffline Minion tasks [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14659:
URL: https://github.com/apache/pinot/issues/14659#issuecomment-4384248688
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Bug] Data Inconsistency issues in RealtimeToOffline Minion tasks [pinot]
Posted by "Harnoor7 (via GitHub)" <gi...@apache.org>.
Harnoor7 commented on issue #14659:
URL: https://github.com/apache/pinot/issues/14659#issuecomment-2542245290
Since created offline segment names for offline table will be same, segments just gets replaced. But if user modifies task config, segment names can change and then we get incorrect query results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [Bug] Data Inconsistency in RealtimeToOffline Minion tasks [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14659: [Bug] Data Inconsistency in RealtimeToOffline Minion tasks
URL: https://github.com/apache/pinot/issues/14659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org