Re: [I] Is pinot possible to support batch Ingestion with Upsert? [pinot]

["rohityadav1993 (via GitHub)" <gi...@apache.org>]

rohityadav1993 commented on issue #10896:
URL: https://github.com/apache/pinot/issues/10896#issuecomment-1962725406

   Tried to upload segments to an upsert table and faced with the following challenges based on this previous [discussion doc](https://docs.google.com/document/d/1STYxZsUYGcYrzHdmOBymQ1NkpxuEzLa4OTiw4mGHiKk/edit#heading=h.2nui75aa5u6b):
   
   1. Generate a segment using `simple` name generator strategy to create a segment using IngestionUtils and upload:
       There are checks in place which expect the segment to be named as a LLC segment:
   ```
       j.l.IllegalArgumentException: Invalid LLC segment name: testtable_1695407400000_1695414594000_0_1
           at c.g.common.base.Preconditions.checkArgument(Preconditions.java:210)
           at o.a.p.c.utils.LLCSegmentName.<init>(LLCSegmentName.java:42)
           at o.a.p.c.u.PeerServerSegmentFinder.getPeerServerURIs(PeerServerSegmentFinder.java:64)
           at o.a.p.c.d.m.r.RealtimeTableDataManager.downloadSegmentFromPeer(RealtimeTableDataManager.java:604)
           ... 15 common frames omitted
       Wrapped by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid LLC segment name: testtable_1695407400000_1695414594000_0_1
           at o.a.p.c.d.m.r.RealtimeTableDataManager.downloadSegmentFromPeer(RealtimeTableDataManager.java:612)
           at o.a.p.c.d.m.r.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:539)
           at o.a.p.c.d.m.r.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:420)
           at o.a.p.s.s.h.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDa...
   ```
   2. Use `fixed` name generator strategy and name segment as LLC name:
   ```
   Got invalid segment metadata when adding segment: rta_rider_sessions_test_REALTIME__0__0__19700101T0000Z for table: testtable_REALTIME, reason: New uploaded LLC segment must have start/end offset in the segment metadata
   ```
   
   We may have to solve this by considering below options:
   
   1. Allow offline segments like names in real-time table
   2. Allow LLC segment like names for uploaded segment without enforcement of offset metadata. (the client needs to manage segment name conflicts)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org