You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <ap...@gmail.com> on 2021/08/24 02:00:20 UTC
Apache Pinot Daily Email Digest (2021-08-23)
### _#general_
**@humengyuk18:** Hi All, anyone know what’s the buildQueryPlan_Count metric
mean in pinot server?
**@xiangfu0:** This is a timer to measure the latency of query phases,
`buildQueryPlan` is just one phase. `buildQueryPlan_Count` is published as the
count of the metrics, you can ignore it as you already have query count
metrics
**@xiangfu0:** See `org.apache.pinot.common.metrics.ServerQueryPhase.java` for
server query phase references
**@humengyuk18:** I’m seeing thousands of buildQueryPlan_Count from server but
only very few query count from broker, is this normal?
**@mayanks:** Hmm, for a single server, the count should match the broker.
However, if there are multiple servers the aggregation across servers would be
large. How many servers do you have?
**@grvmakkar:** @grvmakkar has joined the channel
**@gregsimons84:** Hi all is this still the right place for doc updates ?
I’m checking as I can see inconsistencies between what is published and what’s
in the source
**@kulbir.nijjer:** Yes that should be correct place: There is also
<#C023BNDT0N8|pinot-docsrus> channel that you can join.
**@gregsimons84:** Thanks I will join. Issue I saw was content here doesn’t
match what’s in here but I’ve found the right file now. Looks like some old
files to clean up. Thanks I’ve got some changes il push in review later
**@kulbir.nijjer:** Thanks @gregsimons84. cc @mayanks
**@mrpringle:** How are people handling timeseries analytics with pinot?
Looking for things like zero filing with last value (step interpolation),
summing across interpolated series to get a total by time bucket, get last and
first value in a time bucket, moving averages etc. Guess we can go over to
python etc and implement but it would be nice for some native analytic
functions.
**@gregsimons84:** I’m not well placed on this one but i was experimenting a
little with Apache third eye and there are some percentage based filtering
and some based around holt winters
**@g.kishore:** Most end up implementing udf to achieve that.. if you think
some of these are useful beyond your use case, please file issue
**@mrpringle:** Looks like presto has a pretty broad set of aggregation and
window functions. In pinot can we implement aggregates as udfs?
**@g.kishore:** Yes
**@luisfernandez:** @luisfernandez has joined the channel
**@s.swaminathan:** @s.swaminathan has joined the channel
**@art.artik:** @art.artik has joined the channel
### _#random_
**@grvmakkar:** @grvmakkar has joined the channel
**@luisfernandez:** @luisfernandez has joined the channel
**@s.swaminathan:** @s.swaminathan has joined the channel
**@art.artik:** @art.artik has joined the channel
### _#troubleshooting_
**@anu110195:** Hey , was trying to ingest s3 hosted json file using rest
api, but getting some exception in controller node
**@anu110195:** ```2021/08/23 05:26:07.465 ERROR
[PinotIngestionRestletResource] [jersey-server-managed-async-executor-0]
Caught exception when ingesting file into table: networks3_REALTIME. Cannot
ingest file into REALTIME table: networks3_REALTIME
java.lang.IllegalStateException: Cannot ingest file into REALTIME table:
networks3_REALTIME at
shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestData(PinotIngestionRestletResource.java:184)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestFromURI(PinotIngestionRestletResource.java:170)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_282] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_282] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_282] at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_282] at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:381)
~[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:819)
[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-
all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-
all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-
all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-
all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-
all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:814)
[pinot-all-0.7.0-SNAPSHOT-jar-with-
dependencies.jar:0.7.0-SNAPSHOT-7bcbee5058c6a56d02b2d34e173185009ac35146] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_282] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_282] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_282] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_282] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]```
**@mayanks:** ```.checkState(tableType != null, "Must provide table name with
type suffix for table: %s", tableNameWithType);```
**@anu110195:** I'm sending type as well, (networks3_REALTIME)
**@anu110195:** is there any way to import data in realtime table apart from
realtime ingestion via kafka
**@mayanks:** You can use other steam systems like Kinesis. Could you share
your requirement a bit?
**@nadeemsadim:** what should be the disk allotment for pinot dev
installation.. means for controller, broker , minion .. the disk space
allotted for server is 200 GB * 3 pods = 600 GB total .. so what should be the
controller , broker , minion disk requirement in proportion to that for 3 pods
each on k8s .. @mayanks @xiangfu0 @g.kishore @ssubrama @jackie.jxt
**@kangren.chia:** +1 would love to see more info on server/broker/controller
sizing and tuning
**@nadeemsadim:** also is ssd recommended only for pinot server or for
everything .. also if we are not loading offline segments .. do we really need
3 pods of minion .. and if ram is 8gb * 3 = 24 gb for pinot server.. for
broker, controller , minion . .what should be the ram allotment .. same goes
for cpu cores if cores for server is 2.5 cores per pod * 3 = 7.5 cores .. how
much core allotment will be prudent for controller , broker ,minion
**@mayanks:** Controller only needs storage to store the pushed segments.
Deepstore is recommended for this, not local storage. broker does not need any
local storage either
**@mayanks:** SSD is recommended for Pinot server only
**@nadeemsadim:** what about minion
**@nadeemsadim:** and RAM allotment for broker/controller/minion?
**@mayanks:** Minion as well. CPU/Mem depends on the workload you want to
throw at the cluster
**@nadeemsadim:** is the workload real time data ingested or offline data
loaded .. I mean minion is involved in real time ingestion also or only while
loading data into offline tables
**@grvmakkar:** @grvmakkar has joined the channel
**@luisfernandez:** @luisfernandez has joined the channel
**@s.swaminathan:** @s.swaminathan has joined the channel
**@art.artik:** @art.artik has joined the channel
### _#pinot-dev_
**@xiangfu0:** Hi Pinot devs, I’ve finished this issue to clean up our code
repo to reformat the code and fix the checkstyle errors and enabled checkstyle
maven plugin. There are a few slight changes of the coding style, please re-
import the intellij checkstyle xml file. Still working on eclipse xml.
### _#getting-started_
**@luisfernandez:** @luisfernandez has joined the channel
### _#pinot-docsrus_
**@snlee:** @snlee has joined the channel
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org