Apache Pinot Daily Email Digest

[Pinot Slack Email Digest <ad...@gmail.com>]

<h3><u>#general</u></h3><br><strong>@afilipchik</strong>Thank you!<br><strong>@afilipchik</strong>When I tried it it didn’t convert, just failed with exception.<br><strong>@jackie.jxt</strong>`APPEND` push type will enable the retention manager to delete old segments automatically based on your retention config<br><strong>@jackie.jxt</strong>PERCENTILEEST uses the QuantileDigest data structure to estimate the percentile, which only takes integral numbers. floating numbers will be converted to integral numbers. For floating numbers, you can use PERCENTILETDIGEST instead.<br><strong>@elon.azoulay</strong>Nice, thanks for the info!<br><strong>@fx19880617</strong>This access mode on gcp is only available as nfs <br><strong>@elon.azoulay</strong>Got it, thanks<br><strong>@fx19880617</strong>Cause for pint controller to serve segments, you want a volume mounted to all the pods, which is ReadWriteMany <br><strong>@elon.azoulay</strong>And just one more n00b question (still new to pinot) - what's the advantage for using gcs for controller deep storage?<br><strong>@elon.azoulay</strong>Thanks a lot! Sounds like this should be enough to get started.<br><strong>@fx19880617</strong>This should be sub directory under your gcs mount<br><strong>@fx19880617</strong>Yes<br><strong>@elon.azoulay</strong>parameter?<br><strong>@elon.azoulay</strong>Is that the

```controller.data.dir```<br><strong>@fx19880617</strong>You need the change the volume <br><strong>@elon.azoulay</strong>Yep<br><strong>@fx19880617</strong>Yes <br><strong>@fx19880617</strong>Also note that this is per server per gce disk<br><strong>@elon.azoulay</strong>Nice. We have pvc's working right now - but for the controller to use the gcsfuse mount what config parameters would I need to set?<br><strong>@fx19880617</strong>In helm, you can change storageclass for Pinot server to get this <br><strong>@fx19880617</strong><https://u14005206.ct.sendgrid.net/wf/click?upn=62ZDhgpSpn-2FFZJYORQMk1Y3LBcTKJy4lgdVk6OC3vM9qYlVz8GrqO-2FblCVzsIADi-2BsuBZ7rImtaLMCojYAxpy-2Flj2HAAv-2Bex-2FJ2vJCUGK3bS1s0DPeDXPs2rQcPmWq1W_muK6fcxu-2F-2B2-2BTUDJzQlBQr5sBO533KDMRLp4pQVc1XJu1WMxQYLrdfJaxEW4vCsjclz-2B3JQG-2FDweCGNonhof6zucPrLcmYPytUAdlzfD1E8v7rDZOqlrZlRy0GMkif1lCKBWBBOWMMq39TEI5EEwqRvwMwGjh669noN72tn0p0fcOpL3PgZ4rsEVb2rhmo2a-2B3bf7op7oaCzznfl8m7wRXmwdw748u5AaDfsQVykvPA-3D> this is the  disk for Pinot servers <br><strong>@fx19880617</strong>Yes <br><strong>@fx19880617</strong>For server we suggest to mount persistence disk <br><strong>@elon.azoulay</strong>The second option I mean<br><strong>@elon.azoulay</strong>I'm not sure I know how to do that one, the link you shared describes how to do that?<br><strong>@elon.azoulay</strong>Ah ok<br><strong>@fx19880617</strong>The other is that you maintain a nfs service and deployment which mount gcs then your Pinot controller pod just need to mount a nfs <br><strong>@elon.azoulay</strong>Ok, and that's for the controller pods, right?<br><strong>@fx19880617</strong>There are two ways, if you can build your own image, then you can have one pod which mount gcs during pod init <br><strong>@elon.azoulay</strong>Nice, I'll try this out. This is better than using the HadoopPinotFS with gcs?<br><strong>@fx19880617</strong><https://u14005206.ct.sendgrid.net/wf/click?upn=62ZDhgpSpn-2FFZJYORQMk1Y2K5dfwnhQEYj0NfDJxsP08O5y-2FhVZCMIrK2K8PeBde-2BD1XpfOov4KD1CCV-2FjmRq9j-2BHiHq4rzK1RYme0wJpwGFEP4QTm2pG17l-2FnBWh6dwBuQfiP27In5k1ROl74wsQQ-3D-3D_muK6fcxu-2F-2B2-2BTUDJzQlBQr5sBO533KDMRLp4pQVc1XJu1WMxQYLrdfJaxEW4vCsj4nKg-2BgeFJX-2B7Lq7hIU0DhYwZRU23IcWE7wcm8lJqoMjLghHpyjaEzAX2udN6b10YxIGTiP-2F-2FrbSaY7lWJIKIhjkA04Yw2C-2BuPYlBVkgfRhy50N46e-2FXRkqgjcTC0rBzYfondW2Epe39ygShX7fR-2FV-2FXPascMOxEvIvwDstv7iaE-3D><br><strong>@fx19880617</strong><https://u14005206.ct.sendgrid.net/wf/click?upn=62ZDhgpSpn-2FFZJYORQMk1cp-2FaGB5bPWjrLEHwEDCL-2FvV5JyPayUQGB9qjNpjds-2FETKgZYRYmKdTdynjotHIgRQ-3D-3D_muK6fcxu-2F-2B2-2BTUDJzQlBQr5sBO533KDMRLp4pQVc1XJu1WMxQYLrdfJaxEW4vCsjJHFN6h8Sz-2BgaGQ5GP3ukmtDKjHye3jHgK5CNI-2B3QuuEjIFOEvsIlEqJK-2FS9eyv5HU578C1-2B8pvDs-2FnLMpAo8-2BbZjoVksX0Cg6mOQ-2Faf3huxbDVOhO-2BmFt0YOBzt0psFHGAfZnq8B4TBKHlMIZIy8rP3Th1iZhIO2euAgXFWF-2Fec-3D><br><strong>@elon.azoulay</strong>Thanks for the advice!<br><strong>@elon.azoulay</strong>Ah I see - you add it to the image and then run gcsfuse as part of the start up?<br><strong>@elon.azoulay</strong>So you would use the gcsfuse utility in the helm chart (init job?) to mount the bucket to the path?<br><strong>@elon.azoulay</strong>A cool, I just got it<br><strong>@fx19880617</strong>There is an open source repo of gcsfuse <br><strong>@fx19880617</strong>Then mount the nfs to all controller pods <br><strong>@elon.azoulay</strong>Nice! How do you do that? Or is there a pull request or code you can point me to?<br><strong>@fx19880617</strong>The thing I’ve tried out is to make a nfs on top of gcs<br><strong>@elon.azoulay</strong>Ah, makes sense - so how would we use gcs for controller deep store?<br><strong>@elon.azoulay</strong>For the temp directory or are you saying we shouldn't use gcs buckets at all?<br><strong>@fx19880617</strong>gs could be used as controller deep store<br><strong>@fx19880617</strong>I suggest to use persistent disk for Pinot server<br><strong>@elon.azoulay</strong>Hi, yes, that's what I mean:)<br><strong>@fx19880617</strong>Do you mean use gs buckets?<br>