You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2020/08/26 02:00:13 UTC
Apache Pinot Daily Email Digest (2020-08-25)
<h3><u>#general</u></h3><br><strong>@sanjay982: </strong>@sanjay982 has joined the channel<br><strong>@nanda.yugandhar_slack: </strong>Is there support for Piont with existing parquet files and if it supports, can it build it's own indexing on top of that ?<br><strong>@antonbas: </strong>@antonbas has joined the channel<br><strong>@kennybastani: </strong>Hi all! The one and only @npawar is presenting on Pinot and Kafka at the virtual Kafka Summit conference. If you signed up for free and can access the talk live, please tune in and support one of our own. <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMd9SwS4DIxFqqdjhpJqSqaIR4rpL-2Ft612EamwBRh9cmAU-2FC5F2Gs-2Fn0AQQqoJLqE-2Bx4vp2kIBTNYdI3SqwNibI4-3DDXVr_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mA1oUUfk935HA-2BlgoeHU9zsyDURV8bGyhadjZYRdfzMCYRflX8-2FfX-2BGNh62idlovKb7TpZMIU9-2B82-2BM7QewExdz-2FJKrNmzc8GbidKsVY6lrxutynr-2BdBeEBQzdrPjavBV-2BcnHf4jbJLEhe-2BWiKPYi0hKkjnmGj6WZAAtmlTEqTsE8-3D><br><strong>@karan.nuts: </strong>@karan.nuts has joined the channel<br><strong>@sjeetsingh2801: </strong>@sjeetsingh2801 has joined the channel<br><strong>@respergu: </strong>@respergu has joined the channel<br><strong>@respergu: </strong>hello<br><strong>@g.kishore: </strong>Hi @respergu<br><strong>@slachiewicz: </strong>@slachiewicz has joined the channel<br><strong>@adpolak: </strong>@adpolak has joined the channel<br><strong>@adpolak: </strong>Hi Everyone :slightly_smiling_face:<br><strong>@vkryuchkov: </strong>@vkryuchkov has joined the channel<br><strong>@vkryuchkov: </strong>Hi! I'm learning about Pino and there is one thing I can't find in documentation - is it possible to get data from Kafka which allows only SASL_SSL connection? Thank you.<br><strong>@g.kishore: </strong><https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMdTeAXadp8BL3QinSdRtJdo5Vml3IIBLs7JiV3aA7xH6E1xuyunoVEcnqa9dbWn9CJtee9zt1zS2vZW4Bc-2FI7OT2S7h91HyM56Tdn3p1HYrC-2FFNkxhJCbpPGS2bgYqmgt1Bi9rriOfKdQEfzJMhXD-2FMT-2FKSrHoBaJBwmPvFT9MjQM1eY_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAggDl4hb9nIJaFhn6wexIsHL0sNOb0NPhoENZVqhnIJdpt1hLPxgzbm2xCzfBQdfemwsruhJXs-2ByrzeFNBH0XvCAg3DRQeVplwAlv1-2BluLhOjtc8KglpkU6Xd158XYUx11KPWUPL6-2BSV9g1YMQDh8wZVNbZzACB3fYSQfeqQGo0c-3D><br><strong>@g.kishore: </strong>Use Kafka Partition(Low) Level Consumer with SSL<br><strong>@g.kishore: </strong>what ever is in streamConfigs section will be passed to KafkaConsumerProps<br><strong>@vkryuchkov: </strong>@g.kishore thank you! Will start from there.<br><h3><u>#random</u></h3><br><strong>@sanjay982: </strong>@sanjay982 has joined the channel<br><strong>@antonbas: </strong>@antonbas has joined the channel<br><strong>@karan.nuts: </strong>@karan.nuts has joined the channel<br><strong>@sjeetsingh2801: </strong>@sjeetsingh2801 has joined the channel<br><strong>@respergu: </strong>@respergu has joined the channel<br><strong>@slachiewicz: </strong>@slachiewicz has joined the channel<br><strong>@adpolak: </strong>@adpolak has joined the channel<br><strong>@vkryuchkov: </strong>@vkryuchkov has joined the channel<br><h3><u>#troubleshooting</u></h3><br><strong>@yash.agarwal: </strong>I am getting error in prestodb when running the following query.
```select channel,
sales_date,
sum(sales) as sum_sales,
sum(units) as sum_unts
from pinot.default.guestSlsLitm
where channel = 'WEB'
group by channel, sales_date
union all
select channel,
sales_date,
sum(sales) as sum_sales,
sum(units) as sum_units
from pinot.default.guestSlsLitm
where channel = 'STORES'
group by channel, sales_date;```
Each of the individual queries work separately. but the union does not.
Even the explain plan fails with
```Query 20200825_180515_00000_mshj8 failed: Expected to find the pinot table handle for the scan node
com.facebook.presto.spi.PrestoException: Expected to find the pinot table handle for the scan node
at com.facebook.presto.pinot.PinotPlanOptimizer.lambda$optimize$0(PinotPlanOptimizer.java:86)```<br><strong>@dagrawal: </strong>@dagrawal has joined the channel<br><strong>@pradeepgv42: </strong>QQ, I am using following `ingestionConfig` for my offline table, but I don’t see `epochMinutes` getting populated.
Is ingestionConfig only applicable to realtime tables?
```"ingestionConfig": {
"transformConfigs": [
{
"columnName": "epochMinutes",
"transformFunction": "toEpochMinutes(timestampMillis)"
}
]
}```
<br><strong>@pradeepgv42: </strong>I am using spark executor to generate and publish the segments into pinot<br><strong>@fx19880617: </strong>I think it’s only works for realtime right now @npawar ^^<br><h3><u>#pinot-0-5-0-release</u></h3><br><strong>@ssubrama: </strong>Are people updating the release docs after each release? Some things change over time, and it is best to update the doc or flag something if it is not right or has changed. I think 0.5.0 took longer for credentials. It is best to get the credentials straight for _all_ committers, even if you are not planning to do a release soon.<br><strong>@g.kishore: </strong>I see <br><strong>@fx19880617: </strong>My feeling that the most time consuming part for release is: 1. running the test; 2. update the license and dependencies<br><strong>@fx19880617: </strong>it would be great if we can automate those<br><strong>@tingchen: </strong>The tests for 0.5.0 are done now. I am working with Xiang on the release license update -- it looks like the runbook is a bit outdated.<br><strong>@tingchen: </strong>One main question I have is about the modules we should include in pinot-distribution. It looks like currently Pinot-distribution takes the minimalist approach when it comes to module dependency.<br><strong>@tingchen: </strong>For plugin modules, should they be included in the Pinot-distribution?<br><strong>@tingchen: </strong>@g.kishore @ssubrama @fx19880617<br><strong>@g.kishore: </strong>lets include them for now<br><strong>@fx19880617: </strong>@tingchen i think you can include when running the command to resolve the license and dependency<br><strong>@fx19880617: </strong>but not in real release<br><strong>@fx19880617: </strong>when building the distribution<br><strong>@tingchen: </strong>will do. I can add it as one-off manner w/o merging them to master.<br><strong>@fx19880617: </strong>right<br><strong>@fx19880617: </strong>I only see it useful when generating the license<br><strong>@fx19880617: </strong>however this is very manual step<br><strong>@tingchen: </strong>in future, we should probably uses a different way than pinot-dist to generate license info.<br><strong>@fx19880617: </strong>maybe we can have a module just for pinot-license and add the corresponding stuffs<br><strong>@fx19880617: </strong>so we can programmatically get the license updated<br><strong>@ssubrama: </strong>We should keep a branch in which we can checkin the changes to pom files, etc. for cutting the release. We can add instructions to cherry-pick (with `-n` maybe) these exact commits at various steps. This will reduce the manual steps of cut/pasting a diff or manually editing pom files. It will also make the folks update the doc if the cherry-pick does not merge well, create a new cherry-pick that has the right intention<br><strong>@fx19880617: </strong>agree, if we can change them into multiple release step scripts, then that would be great<br><strong>@fx19880617: </strong>like `release_helper/step_0.sh`<br><strong>@ssubrama: </strong>I don;t know how to automate the license part even more. Before I took on, it was "compare what you see on the web with what you see on the file and update". I changed it to a bunch of awk/sed scripts to extract and sort etc.<br><strong>@ssubrama: </strong>If we can get an html scraper than we are better on that front. I didn't know how to do that.<br><strong>@tingchen: </strong>the part on deleting obsolete binary from current binary file can be further improved using a script.<br><strong>@tingchen: </strong>But the main issue now is that we can not get full deps from the master branch's pinot-dist pom.xml.<br><strong>@tingchen: </strong>There are other ways to get the full dep list from a Maven project though. Sth along the line with <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSEKVzVIUMC6rhYqnFmZbQm61rdNHxELiNHwBel16jgXy5BCZesJ12tzWD2m2jjLaIlDduvQUkBrAGwedAH4sFQXBDwkysBuA7LXyxZMQNjaecJwXkDssouVBc6TpsLaF1yd-2FFp3GTrHWqlCgtPG9PendZkoWWAEiqsMf0BuqIPZB64B_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAU6rsqJrXbty1U8C1dOnabwq3oq7e4AMyJX-2BpDLbnySHJjhmqKmha97wdJ-2Bzkcbfr-2FTHnweQjQQD4fYVsVlNQ7NN4y-2BGHqbZ0nk7FXHPI-2F0zVJmWdquXZ2Ypqh4N9b0GJC-2BVQ2h8ZIWx8h1VJuH6rNnbNehDhfQNgYcBvyG8X91Y-3D><br><strong>@tingchen: </strong>not sure if you guys have consider that before.<br><strong>@tingchen: </strong>Added the plugin modules and revised <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSfW2QiSG4bkQpnpkSL7FiK3MHb8libOHmhAW89nP5XK4rP-2BkFe5YEFfdRMoaAM6kg-3D-3DlQkz_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAHTfaUS6G8SP5P27nx-2BbxqEb3Dp2CMDvkC1mf1MLGJAIq10447man-2BHPYlM8AiP-2FM9clBDwQqsDolVKbqgacB6EKL2mOBakodf6YQKfN1fqCnK0GWCjWGn4GIQcLCy3d1e0DqgyH2-2FH0ml5-2BWUvVHM-2BhrGGrvVKA5Ni-2FwKgTOjUg-3D>. @fx19880617 and @ssubrama can you take another look?<br><h3><u>#lp-pinot-poc</u></h3><br><strong>@fx19880617: </strong>@andrew what’s your hardware set up for pinot?<br><strong>@fx19880617: </strong>I wanna reproduce the latency scenario<br><strong>@andrew: </strong>You can see this in `pinot-helm.yaml`
<https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMSpQpf1vBNN7VUyMr5oI-2BtCAcH-2BA3iHH8tMnNcU0bwQJ7jx5-2B09UtwlYTMtvS0x-2BsWWDyg5OHT4JMkxNKk3qFa9S7R6zQTpdWCUu19XIZMqa9x3R_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mA3FaP80lg-2FbW467tuQhvpMvR61dZXni177nGwYqo1PUg7na-2F7nYsIlUytE6d8Y8mmiI8-2Fr50kku4s7lS0UWJE8ELOlcgGMM9ebnRDFf-2FhxmpASqT0YEeovcek-2BjQyPtDWB3qJy-2BfmfkoTZz0gdX0Vgrht3HjL0EhVT95UYmZjF6I-3D>
It’s running on GKE on n1-standard-16 nodes, and persistent SSD for data storage.
Does that provide everything you need?<br><strong>@fx19880617: </strong>looks good<br><strong>@fx19880617: </strong>many thanks!<br><strong>@fx19880617: </strong>so totally you have 18 nodes running?<br><strong>@fx19880617: </strong>seems that we need 4 nodes for controller/brokers and 16 nodes for servers<br><strong>@g.kishore: </strong>@fx19880617 were you able to reproduce the tail latencies?<br><strong>@fx19880617: </strong>i need to setup the cluster first<br><strong>@fx19880617: </strong>will do it today<br><strong>@g.kishore: </strong>ok<br><strong>@andrew: </strong>in general, yes although the cluster has a few other things running (on staging, so low utilization), so it probably spreads across more GKE nodes<br><strong>@fx19880617: </strong>got it :wink: I will try to replicate the similar environment. Since you have set request/limit resources to be the same, it should be ok<br><strong>@fx19880617: </strong>i’m seeing some issue with this exception:
```[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.5.1:build (default-cli) on project ingest-api: Build image failed, perhaps you should make sure your credentials for '<https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYdUo-2FB9-2FZiFxfreLUjKLukT3ncsWCvnohdqxk3CFRnlyMBOg6PxGNDCPKMPyJVevbIm9pavZRy0O45YuKAzmElS3WADmBv-2BcsHhUEgUsQohYU3CHzM0U9dkCOGAz6JIOdJ1eg7E2FYl6FfPfg-2BxCDetI9OX3-2FNAfjxloUPAM-2FYVDxPAj53QElbMzUoO5Tmpf4A-3D-3DWR_h_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAxrOLujWP9EPRRYZPF68yW2aGw5iM020DCrJgMM8yRCOiKQBe83BgqrYssQLpb1F8nl5EDCjdDV8QMhcUR6xtb2x1xdFaZwuumUXrL1xw1zrHzR-2F0hXfzHptLwotjZghIMUafnUgK87wdnBNBLK7T0Wss3cRQhW3EGOkuycfJjG8-3D>' are set up correctly. See <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMXt7LSzQUx5TiOrxtuA-2FdDGB0KK7lePNNc7YwQh-2B51TFVIcTxIrQ-2BeItTYWAQmKwz-2F6XkthyotKxt807kS2ZVzugf2XD-2BdY9ZBzyF05sa4hCJHrDBeeBx0OTPy72iHvLi8NKIn3W1ocmn3qONbHdhCGkTcu2zdvunqT7cz-2FSC2uQfXxg_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAfE0Rkjqfjk2BDYoCO62YD3LnliuQhgkmSWP3o5nVBbu3RFijMCMtWswQIUxQdSdCbQ4Z2H-2B24KdLNNfh7BbslCef9tZ2VGzaldnyIsgOJqDxIvytfbItEDY1dc7BQhGhSpLt7pGyqpMMKbvRqGTINr63PgURe3sm-2Ff-2BkGJyNZIM-3D> for help: Unauthorized for <https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYdUo-2FB9-2FZiFxfreLUjKLukT3ncsWCvnohdqxk3CFRnlyMBOg6PxGNDCPKMPyJVevbIm9pavZRy0O45YuKAzmElS3WADmBv-2BcsHhUEgUsQohYU3CHzM0U9dkCOGAz6JIOdJ1eg7E2FYl6FfPfg-2BxCDetI9OX3-2FNAfjxloUPAM-2FYVDxPAj53QElbMzUoO5Tmpf4A-3D-3DBq44_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mABUH-2FcMVjQJSGT0EYn2I5qr0P3qmtHjVXO5yJeoherCmoGqviTbRFTMPo7KAkXi3a7vw7ddM6xEK3dBA4YhOTcQVgEhv0Q57Lbf6sSJBr3NZGBUjInHsAIL1MBdqqua3EgMP8ne-2Fu9XXwnFyg8nTkbPsWdFODL3znJ0Tpe5UpfZs-3D>: 401 Unauthorized
[ERROR] {"errors":[{"code":"UNAUTHORIZED","message":"You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMcdN0SE1Hov52ehRdpfp8kgcoByfWGwl8P9nItU7iu3Pr3h3IOxw7u5c4E-2FjueLh71-2Bny8-2FYsY3e7nzZITEsbN3zdBMAy3EcqG-2B2EYzuyJAh3mPG_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAcem8oQqE9ls9yDm-2BwAtkN8EURtOvizOf1px9Qo7vRUZe9jvQJaPXfjxkW97AGIjNRXaKA1wwQyQS-2FDACu2KQIYQsLQyBGKbcuBsRtFEXTmpmftlfqK69yGGyIQCQGRvIEu-2BRp1U-2FoY2wS31-2F-2BsecqbD22Om0HAtDaYMZ4-2Bf65a0-3D>"}]}```
<br><strong>@fx19880617: </strong>I guess this requires credential to access your internal stuffs<br><strong>@fx19880617: </strong>this is when I try to run `mvn compile jib:build --projects DataIngestion/IngestApi -DskipTests`<br><strong>@fx19880617: </strong>```+ mvn compile jib:build --projects DataIngestion/IngestApi -DskipTests
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.leanplum.pinot.dataingestion:ingest-api:jar:1.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.springframework.kafka:spring-kafka:jar -> duplicate declaration of version (?) @ line 160, column 17
[WARNING] 'build.plugins.plugin.version' for org.springframework.boot:spring-boot-maven-plugin is missing. @ line 174, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ------------< com.leanplum.pinot.dataingestion:ingest-api >-------------
[INFO] Building Ingest API 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ ingest-api ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ ingest-api ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- jib-maven-plugin:2.5.1:build (default-cli) @ ingest-api ---
[INFO]
[INFO] Containerizing application to <https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYdUo-2FB9-2FZiFxfreLUjKLukT3ncsWCvnohdqxk3CFRnlyMBOg6PxGNDCPKMPyJVevbIm9pavZRy0O45YuKAzmElS3WADmBv-2BcsHhUEgUsQohYU3CHzM0U9dkCOGAz6JIOdJ1eg7E2FYl6FfPfg-2BxCDetI9OX3-2FNAfjxloUPAM-2FYVDxPAj53QElbMzUoO5Tmpf4A-3D-3DYsDj_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAhd76hEWseCGw1ykqBWKjR5I8VyHyyTQg5YGv6v58nvA-2F3JprkJd3ADMPjS3LZ3809qp5Dn-2Fhzov1Hm9OAgJZQj9BLYzrW6SUYWDWWvyhqNGi9F1resLiOAkL-2FzAu8O04gUG3ZR-2BEBLrGuZQBapXv49xn-2FeVwzLI2G0x3RFMoQDE-3D>...
[WARNING] Base image '<https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYeBFwR5fuM7uyT7U0xTbqJ7baDGysjonDOYeZA8n9fCtUY1uTnh7-2BJhCbAAIT3KBOtSPy6wBlv4tK8Q3YJHicfU-3D8ko2_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mApmjNyddD-2BFfv0erfMhAfa1VXjEMh-2FL9vxnnXxWN9ZZXbtKQGiqefxwL2Y-2FjKQHKISEqFiVj2FKpXNUJmP-2BGmiDCmFnwbetP3lmNilxQroKdUjFPu4lKxJ-2BCa2wKRWrO3Onrhs7z92WahoEfEKSUJJE8BRPASrIQV6H2b6V-2BJF2o-3D>' does not use a specific image digest - build may not be reproducible
[WARNING] The credential helper (docker-credential-desktop) has nothing for server URL: <https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYfdg2AJPi3h5haXXzW2xsvk-3Dcymj_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mA-2FD-2Fo0ix5FOcoNtfzZRuO4MzpQ9qFoIBGYSVKwaVIO6O1iX8O2sBuOESdbut65hxk3t1YT2bkvu7QRZkKqnL7y55LC7Vo9xEpCUEfL-2BsLjLImGR9MMpJE2CKyAxaxpHayU5rJXh6g2J6QuxuLvl8VxMhUQ8BLaa5btOuCapJm9PE-3D>
[WARNING]
Got output:
credentials not found in native keychain
[INFO] Executing tasks:
[INFO] [== ] 6.7% complete
[INFO] Using base image with digest: sha256:de50b102c008835736d045adf71620a9330a079bd224417afade620d279e90f4
[INFO] Executing tasks:
[INFO] [======== ] 26.7% complete
[INFO] > building dependencies layer
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.596 s
[INFO] Finished at: 2020-08-25T15:33:37-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.5.1:build (default-cli) on project ingest-api: Build image failed, perhaps you should make sure your credentials for '<https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYdUo-2FB9-2FZiFxfreLUjKLukT3ncsWCvnohdqxk3CFRnlyMBOg6PxGNDCPKMPyJVevbIm9pavZRy0O45YuKAzmElS3WADmBv-2BcsHhUEgUsQohYU3CHzM0U9dkCOGAz6JIOdJ1eg7E2FYl6FfPfg-2BxCDetI9OX3-2FNAfjxloUPAM-2FYVDxPAj53QElbMzUoO5Tmpf4A-3D-3DzmDv_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAtwUAtilGCopfTcQtUF2XHBmu-2FvVgvI-2BvSE9S8g12YuNXdir8ibQi3AL7IRZAmJOgIf3HZdoPP79DM-2BFyprsNmzUHqlSKl8F1KtXIBbVEeLFfwiuQfW8FJqFQX2xgyjcZgGkvdxdvMCq0haTFeehvWTsE5LAIUDI9kdPiwz5HBXk-3D>' are set up correctly. See <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMXt7LSzQUx5TiOrxtuA-2FdDGB0KK7lePNNc7YwQh-2B51TFVIcTxIrQ-2BeItTYWAQmKwz-2F6XkthyotKxt807kS2ZVzugf2XD-2BdY9ZBzyF05sa4hCJHrDBeeBx0OTPy72iHvLi8NKIn3W1ocmn3qONbHdhCGkTcu2zdvunqT7cz-2FSC2uQ7DPi_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAkOnuxbRjouLEpixqAQPs6-2FGHLSJ36wp08KFX-2BEV-2BKAMcvGsNzYLhZ5oqkt94mrOxEG7ZRdb3ao8Zfw7JZWd-2B-2BJWisgNS9EO3btOE7QInu5XlCwsYGvvpdFTUfCkrqkaH69tqM7SfvMNIHRMhmFVpdWY4Dojyb3SZMZoPzYB8-2Bvo-3D> for help: Unauthorized for <https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYdUo-2FB9-2FZiFxfreLUjKLukT3ncsWCvnohdqxk3CFRnlyMBOg6PxGNDCPKMPyJVevbIm9pavZRy0O45YuKAzmElS3WADmBv-2BcsHhUEgUsQohYU3CHzM0U9dkCOGAz6JIOdJ1eg7E2FYl6FfPfg-2BxCDetI9OX3-2FNAfjxloUPAM-2FYVDxPAj53QElbMzUoO5Tmpf4A-3D-3DeGl5_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAfSM06znfqYVwMr6PbON5VX6Qz0AEOfn-2BqHfhakiibkxNpfvXUxpDSNtS2-2FDLZee6nz704JxIH444krYxtIJPfTtHjrRslbK1mQCdmlY3Rjttj23AY4fQSkc6sms4Cd8L30nUXs-2F9VG8bNoh5hLkAhuidSGpaPspF-2FEc1qTqMIFk-3D>: 401 Unauthorized
[ERROR] {"errors":[{"code":"UNAUTHORIZED","message":"You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: <https://u17000708.ct.sendgrid.net/ls/click?upn=1BiFF0-2FtVRazUn1cLzaiMcdN0SE1Hov52ehRdpfp8kgcoByfWGwl8P9nItU7iu3Pr3h3IOxw7u5c4E-2FjueLh71-2Bny8-2FYsY3e7nzZITEsbN3zdBMAy3EcqG-2B2EYzuyJAhMue9_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mAWLMl1B-2FhdFYmdZg5yHEN8eF7Yp7BEAi9ZsMbJETV-2FbGKcdlPYQT8SQcxz8CUVV77m6RoNTbSXv-2Fw4IAKiiuC-2BNoRAFQd9IFz3vqSW1ri7vGTaEcxhh5NdSj3qh1vuK1Xo-2F5BqHnryAKpFLPDxm7j2oELFR27pbGHEUR42qz8Fe0-3D>"}]}
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] <https://u17000708.ct.sendgrid.net/ls/click?upn=iSrCRfgZvz-2BV64a3Rv7HYeAX6oy4izXM0HiNP0TVTLeEbZMtnlockE2n83qpngKGzb8Qcxw9RKFxEcelgdutLEUKKX2zy6KGoCdoPMUfizRq-2BTGtI0tFQqvqLaGXBjPL8J97_vGLQYiKGfBLXsUt3KGBrxeq6BCTMpPOLROqAvDqBeTxNjdthT-2FmvrZjtuwsyf6mA6Ei-2FxbnG-2FngaxVE3PrpH0eDge-2BWYcSpfTj68IgN3-2FE-2BoCNwu88YHtSBIcSJnMgnfVGhgxt-2BFVRGZs3GRBn6rTD6kJE8x0Wf8lSDYUR4vaSxJfqLL7E5qyNidcdtNnlaz7z6byOJ-2Fclgn86xmK-2ByrFdyWsHvjBnXdD3IUzyo3M50-3D>```
<br><strong>@andrew: </strong>yeah, some of the settings are configured for Leanplum, so they’ll need to be changed<br><strong>@andrew: </strong>like the image location<br><strong>@andrew: </strong>can you try changing it to a location that you have access to?<br><strong>@fx19880617: </strong>checking<br>