Apache Pinot Daily Email Digest (2020-07-08)

[Pinot Slack Email Digest <sn...@apache.org>]

<h3><u>#general</u></h3><br><strong>@vickykak: </strong>@vickykak has joined the channel<br><strong>@rgarcia: </strong>@rgarcia has joined the channel<br><strong>@brandenc: </strong>@brandenc has joined the channel<br><h3><u>#random</u></h3><br><strong>@vickykak: </strong>@vickykak has joined the channel<br><strong>@alan.hardy: </strong>@alan.hardy has left the channel<br><strong>@rgarcia: </strong>@rgarcia has joined the channel<br><strong>@brandenc: </strong>@brandenc has joined the channel<br><h3><u>#troubleshooting</u></h3><br><strong>@somanshu.jindal: </strong>@npawar I have attached new snapshot of error along with zooinspector snapshot.<br><strong>@somanshu.jindal: </strong>I think commit is failing for me from the start. Generally i had longer commit time or rows so i never checked. I got this issue when i was verifying commit by setting lower values in table config.<br><strong>@alan.hardy: </strong>Hi. I'm standing up a Kubernetes/Helm cluster as per docs. Modified to use S3, again as per docs: <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMdTeAXadp8BL3QinSdRtJdrQSnuJvsa-2BKLAdxKaLYoSk5fnxow1nUtu2wwx55WLnNxUjAtupORoTsl-2B1z4-2FRqPw-3Dys-3_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0Jh1O7ykR84JxmF9yzYD4CwbIHHutAQG6GpnDvObyruJS4x9-2FKDyF7ezgPxFQ7jH39xSlcQloPtSHs1cYzLHfCnrmXn51lv1BzGjsN0oBbjly2RgZDQ-2B-2B-2BoP-2BekLyV10OA-2FSCwSHHbEj6vVw-2Fhkc39l8CzHvyCYOKVrvj4S0PKxc-3D>. Controller &amp; Server are throwing

```ERROR [PinotFSFactory] [main] Could not instantiate file system for class org.apache.pinot.plugin.filesystem.S3PinotFS with scheme s3
java.lang.ClassNotFoundException: org.apache.pinot.plugin.filesystem.S3PinotFS
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_252]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_252]
        at org.apache.pinot.spi.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:80) ~[pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]```
<br><strong>@pradeepgv42: </strong>Hi, I am trying to test following change (<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKwnNfTsLQ-2Fmo2mEKJSpKxog-3D-3DacUg_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0dE4b0jBVzO87Th5SrhtOxGqa-2BXHuoohApXHM9K9CucP2Rxg-2B-2Fbrq7ptiXoD9GEjgz5fmPJAi-2BeVKsUEj2lkEp49q-2B5t-2Bg9Wh3YogiBuAX7JspqyNzXs71ywSDIOIIX9Jbjv-2FRWFTWC-2BI0ShI0w7qX-2B-2FH32r6Qe7GKnOzUH-2B1xsw-3D>) on my cluster. So, pulled code from the master, but I am seeing below exception. wondering if there’s any change you know of? I only see this change (<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XKvd1xlGru5RyM-2Fuil934eiw-3D-3D4UiU_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0xC-2Fk-2B74-2BxM2tPiPqq7Hmz91Ma6EralN7-2BG6So5vCioP-2Bv6Ll-2BBLXAcokwu-2FN2YctK4m-2BjfJvxP-2BZOXC2YRD0BbcmFAEQ2wTM2PGpcr-2BCIBsDhH1rTXyPr7sVLr3AwUxfrNrE1jNRbYL2SlrP62P2nXw-2Fkad8AzXj0Wm9f2KDVdc-3D>) which says that existing behavior  shouldn’t change?

Below is the exception I see, which seems to be trying to fetch the S3 region from configuration
```

java.lang.IllegalArgumentException: null
        at shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:108) ~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependenci
es.jar:0.5.0-SNAPSHOT-2ec7dee1597021742f68f0ae8b279f7560e55894]
        at org.apache.pinot.plugin.filesystem.S3PinotFS.init(S3PinotFS.java:80) ~[pinot-s3-0.5.0-SNAPSHOT-shaded.jar:0.5.0-SNAPSHOT-2ec7dee
1597021742f68f0ae8b279f7560e55894]
        at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:55) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.
jar:0.5.0-SNAPSHOT-2ec7dee1597021742f68f0ae8b279f7560e55894]
        at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:75) [pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:
0.5.0-SNAPSHOT-2ec7dee1597021742f68f0ae8b279f7560e55894]```<br><strong>@mayanks: </strong>As per the code the precondition is `
```Preconditions.checkArgument(!isNullOrEmpty(config.getProperty(REGION)));```
<br><strong>@pradeepgv42: </strong>```pinot.server.storage.factory.s3.region=us-east-2```
I already have this config,<br><strong>@pradeepgv42: </strong>and this was working fine with earlier version<br><strong>@g.kishore: </strong>@dlavoie ^^<br><strong>@dlavoie: </strong>@dlavoie has joined the channel<br><strong>@mayanks: </strong>Yeah, #5608 seems one PR that is related.<br><strong>@dlavoie: </strong>Definately sounds related, I’ll investigate to tomorrow morning!<br><strong>@mayanks: </strong>My guess is subsetting of config is broken.<br><strong>@mayanks: </strong>`
```PinotConfiguration schemesConfiguration = fsConfig.subset(CLASS);```
<br><strong>@dlavoie: </strong>Yeah, that would explain the config being object being null.<br><strong>@mayanks: </strong>Is `class` new?<br><strong>@mayanks: </strong>If so, this is a backward incompatible change?<br><strong>@dlavoie: </strong>We have tests around fsConfit subsetting, if that is broken, it’s definitely not intended. I’m not home right now.<br><strong>@g.kishore: </strong>@pradeepgv42 can you paste the configuration<br><strong>@g.kishore: </strong>entire file<br><strong>@pradeepgv42: </strong>```pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.accessKey=
pinot.server.storage.factory.s3.secretKey=
pinot.server.storage.factory.s3.region=
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.dataDir=/home/ubuntu/pinot/data
pinot.server.instance.segmentTarDir=/home/ubuntu/pinot/segments```
<br><strong>@pradeepgv42: </strong>This is the server config<br><strong>@g.kishore: </strong>tx<br><strong>@g.kishore: </strong>found the issue: <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XK1GA-2FG5rC0yUXuYJG94BOQg-3D-3DTFgm_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0F9V3n9S3OOFZn-2BYLOdN2Fcxtkj2vGB7nZN0Ifivc9-2BlVy49U1OdWPOJvE6L2CYeIrrE5clFe26DuU9urZK7bv79ECYKKU-2BvI8hBnt82Fy1BCbDDoyetXDJn75z9IcsVxoal6COgrYYamUmdwoAeZXQVSKV1bwFYmTSg4P-2FG3MiI-3D><br><strong>@pradeepgv42: </strong>tested the fix, it’s working now thanks @g.kishore<br><strong>@mayanks: </strong>@g.kishore Checked the callers of `init`, going by variable names, some are passing `config` and others are passing `fsConfig`<br><strong>@dlavoie: </strong>I'm back home, I'll review the findings and provide more context if needed.<br><strong>@mayanks: </strong>Thanks @dlavoie. One thing this failure brings up is that we have lack of test coverage here.  Perhaps we should also use this opportunity to improve on that<br><h3><u>#pinot-dev</u></h3><br><strong>@ssubrama: </strong>@dlavoie we had to do this to get a nightly build going. @jlli has more information. Also, why do we need to release a snapshot of pinot for other projects? If you are doing manual builds, can you provide some values for these variables and get a snapshot build<br><strong>@dlavoie: </strong>Having a SNAPSHOT release channel allows a CI pipeline to quickly identify potential breaking changes and new features before the release is published. Most OSS project I have seen have snapshot releases channel.

What I've seen work well is to always have maven version explicit in the poms and manage version upgrade with the mvn release plugin as part of the release process. When a release is cut, snapshot is moved to a final version, tag is done on git, build is published to maven, then main branch is updated with the new version snapshot.<br><strong>@dlavoie: </strong>I feel that after a git clone, `mvn package` should just work without specific instructions.<br><strong>@dlavoie: </strong>Also, when working locally, sub modules of Pinot can't be built individually with `-rf :sub-module` because of that. The whole project needs to be rebuilt.<br><strong>@jlli: </strong>@dlavoie you don’t have to specify anything for revision and sha1 if you want to make a snapshot build; `sha1` has its default value, which is `-SNAPSHOT`. Plus, there is an alternative to make a release instead of using release plugin:
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMZLbmandTFEO1t1g5FzNidEeemdSSybunX9ukF8jlz3roygTevMb76DabYOpwo-2BQaNsZ2WLQVUh4-2FnC-2BcMZ40169EO3-2FjGfvrtKip-2BnK7uMp7BUq_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTzU9ef-2FlazhfOg2CVeWdBR0Uzgn6tFD6X-2F4klGQc5EOsE1I7q-2FQvAKfS7qOx6xwpNGRPrnrAC6BPaHc0M1oJJPUoBJsjVIcyBg-2FadPBzXqv3uaa54UUjQhNpCkrgUx8iF7b87LAghfcryKB-2BWuA-2F1nJZccbMong3XVOWC2eQL5lZTfnGlr-2FeuE9VbJWEVRRXNo-3D><br><strong>@dlavoie: </strong>Hi @jlli default values will only apply if you build from the parent pom. I'll go through that interesting reading. Thank you!<br><h3><u>#presto-pinot-streaming</u></h3><br><strong>@jackie.jxt: </strong>@elon.azoulay The doc looks good.<br><strong>@jackie.jxt: </strong>Can you please address the comments in the protobuf PR? Then we can merge it and start the next step<br><strong>@elon.azoulay: </strong>Sure!<br><strong>@mayanks: </strong>@elon.azoulay posted some minor questions on the doc.<br><strong>@elon.azoulay: </strong>I'll list the future enhancements, thanks for the feedback!<br><h3><u>#s3-multiple-buckets</u></h3><br><strong>@mayanks: </strong>@mayanks has joined the channel<br>