Re: [PR] Add upsert config - outOfOrderRecordColumn to track out-of-order events [pinot]

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on PR #11877:
URL: https://github.com/apache/pinot/pull/11877#issuecomment-1793358161

   > In one of the use-cases where a user was using `skipUpsert` in their queries, they start seeing nulls for a partial-upsert table where previous non-nulls were sent for those columns. On debugging we realised those were because of out-of-order events. Ideally, we had want to give users the feature to skip those out-of-order event records as well in query response when using `skipUpsert`.
   
   Can you give an example of the desired behavior?
   
   > I understand it adds an overhead in case of full-upserts mainly by doing an additional read from the hashmap. For partial-upserts, I think the overhead is mainly around an extra boolean column.
   > 
   > Also one question around this, what is the general recommendation of using `skipUpsert` query options? Should they be only used for debugging purposes or are they reliable enough to use it in a manner for getting all records for a given key (just like an append-only table)?
   
   `skipUpsert` is mostly used for debugging purpose, but can also be used to show the change history. With clean up tasks such as compaction (#6912), change history can be cleaned up though


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org