You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2021/06/25 02:00:19 UTC
Apache Pinot Daily Email Digest (2021-06-24)
### _#general_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
**@sriramdas.sivasai:** @sriramdas.sivasai has joined the channel
**@neilteng233:** Hey, if I want to see the execution plan of a query. Where
can I see that? Is the trace info the execution plan? I have a query like this
```select * from table where
DATETRUNC('day',updatedAt,'MILLISECONDS')>=1624492800000``` and I want to know
whether it use the range index or not on the updatedAt column. ```"traceInfo":
{ "pinot-server-1.pinot-server-headless.analytics-prod.svc.cluster.local":
"[{\"0\":[{\"SelectionOnlyCombineOperator
Time\":1},{\"InstanceResponseOperator
Time\":1}]},{\"0_0\":[{\"ExpressionFilterOperator
Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"PassThroughTransformOperator Time\":0},{\"DocIdSetOperator
Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator
Time\":0},{\"SelectionOnlyOperator Time\":0},{\"ExpressionFilterOperator
Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator
Time\":1}]}]", "pinot-server-0.pinot-server-headless.analytics-
prod.svc.cluster.local": "[{\"0\":[{\"SelectionOnlyCombineOperator
Time\":3},{\"InstanceResponseOperator
Time\":3}]},{\"0_0\":[{\"ExpressionFilterOperator
Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"PassThroughTransformOperator Time\":0},{\"DocIdSetOperator
Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator
Time\":0},{\"SelectionOnlyOperator Time\":1},{\"ExpressionFilterOperator
Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"PassThroughTransformOperator Time\":0},{\"DocIdSetOperator
Time\":0},{\"ProjectionOperator Time\":0},{\"PassThroughTransformOperator
Time\":0},{\"SelectionOnlyOperator Time\":0},{\"ExpressionFilterOperator
Time\":0},{\"RangeDocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"DocIdSetOperator Time\":0},{\"ProjectionOperator
Time\":0},{\"PassThroughTransformOperator Time\":0},{\"SelectionOnlyOperator
Time\":0}]}]" ```
**@gqian3:** Hi, we have a table planned to migrate to Pinot, it has only
dimension fields and one datetimefield, the data volume is nearly constant, we
mainly use it as a look up table, is this a good candidate to migrate to Pinot
to gain any query latency benefit from Pinot for this table?
**@karinwolok1:** Big hello and welcome to all our newest community members!
:wine_glass: @specsek @hsaini @vlum @juraj.pohanka @sriramdas.sivasai
@srikanth @evan.galpin @jiangok2006 @dougdeu @jkim0110610 @leeon2013
@s.azimigehraz @bharath.sbk @huzongxing0826 @hemanga.borah @b.barisercan
@gmr.sql @anindabhattacharjee12 @qun0929 @atri.sharma @thiagopsnfg
@ravinder2021.kr @baluvignesh @aaronlevin @jadami @dang @ojasmulay @mail765
@nithinbharadwajd661 @qianbo.wang @nishanth @hamoop @prasanna.gsl @hashhar
@azri Would love to. know who you are and what brought you here! :smiley:
Also, don't miss our event TODAY with presentations and panels from experts
from companies like LinkedIn, Microsoft, DoorDash, Stripe, Confluent,
Decodable and more! *Starts in 2 hours!* There will also be a great demo of
StarTree Cloud! :tada: :partying_face: We'd also love your support, please
share this event with any of your friends who are also *obsessed* with *SPEED*
and *REAL-TIME-ANALYTICS!* :zap:
**@juraj.pohanka:** Thank you @karinwolok1. Glad to be a part of a very
vibrant community of a project with huge potential! :slightly_smiling_face:
**@karinwolok1:** We're so glad to have you!!!!! :smile:
**@karinwolok1:** See you today?
**@juraj.pohanka:** Definitely :)
**@karthick.nitt:** @karthick.nitt has joined the channel
**@mark:** @mark has joined the channel
**@jai.patel856:** I have some questions about Pinot realtime/upsert tables:
1\. Retention. We have an upsert table whose retention is set to 10 days.
However, I’m seeing “latest” rows where the value is 14 days old. Is the
cleanup process “lazy”? 2\. Although not required are there any
advantages/disadvantages to the time column being the same as the sorted
column? Am I correct to understand that the recommendation is to only have one
sorted column?
**@jai.patel856:** @jackie.jxt
**@jackie.jxt:** 1\. The retention management is on segment level, so the
segment won't be removed if any record within the segment is not expired 2\.
Having a column sorted can accelerate the filtering and benefit from the data
locality. Pinot does not support secondary sorted column yet, so usually it is
hard to sort on multiple columns
**@jai.patel856:** okay, thanks. we’ll keep the sorted index same as the time
column.
**@jai.patel856:** thanks.
**@jackie.jxt:** Based on your use case, you might want to choose sorted
column differently. E.g. if you always filter on a key column (e.g.
`memberId`), you might want to sort on it to benefit from fast filtering and
data locality
**@jackie.jxt:** If you always have a range filter on time, then sorting on
time column is also a good option
**@neilteng233:** Hi, if I do a hot-update on the table config of a real time
table. What will happen to the old index will they be cleaned up
automatically?
**@mayanks:** What kinde of update?
**@neilteng233:** I change one col in the dimensionsSplitOrder of the star-
tree index.
**@mayanks:** Hmm, I am not sure if reload will rebuild the star tree in this
case. @jackie.jxt?
**@neilteng233:** I guess hot update on the forward index and sorted is not
allowed. But I am not sure about the bitmap inversed index, star-tree and
others who are not directly related to the how the data sit on the disk.
**@jackie.jxt:** Yes, the reload should drop the old star-tree and create the
new ones
**@neilteng233:** awesome. Thank you
**@neilteng233:** btw, can I see the trace info as the execution plan of
pinot?
**@jackie.jxt:** You may enable the trace on the query console, and the trace
can be found in the query response metadata
**@jackie.jxt:** It tracks the time spent within each operator
**@ajithreddy118:** @ajithreddy118 has joined the channel
**@canerbalci:** Curious if there are folks from Slack here. Maybe Slack
would be kind enough to donate a Pro tier for Pinot Slack workspace so the
message history could be kept for a bit longer.
**@kennybastani:** Thanks for the recommendation! @srini gave us some extra
details for making this happen. Will check back in the next week.
### _#random_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
**@sriramdas.sivasai:** @sriramdas.sivasai has joined the channel
**@karthick.nitt:** @karthick.nitt has joined the channel
**@mark:** @mark has joined the channel
**@ajithreddy118:** @ajithreddy118 has joined the channel
### _#feat-presto-connector_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#pinot-helix_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#troubleshooting_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
**@patidar.rahul8392:** How realtime backward-Compatible(new column) works in
Pinot. I have some events in Kafka and created one pinot realtime table .next
time I have again pushed some data in Kafka with 1 new field and again
executed AddTable command with updated schema(new column added in schema) and
then on pinot ui did reload all segments.on ui in schema I am able to see that
new column but when I am doing select * from tablename the newly added column
is not showing there. Here I have added city as new column. @jmeyer
**@jackie.jxt:** Please refer to this doc:
**@jackie.jxt:** In order to reload consuming segment, you need to set a flag
in the server config `pinot.server.instance.reload.consumingSegment`
**@jackie.jxt:** It is disabled by default in version `0.7.1`
**@patidar.rahul8392:** I'm able to see other column data only for newly
added column it's not showing any data.this is my configuration file. Kindly
suggest.
**@xiangfu0:** old data are not refilled as pinot won’t consume old data
again.
**@sriramdas.sivasai:** @sriramdas.sivasai has joined the channel
**@syedakram93:** [INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE [INFO]
------------------------------------------------------------------------
[INFO] Total time: 2.094 s [INFO] Finished at: 2021-06-24T19:02:06+05:30
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-
plugin:3.8.0:compile (default-compile) on project pinot-spi: Compilation
failure [ERROR] javac: invalid target release: 11 [ERROR] Usage: javac
<options> <source files> [ERROR] use -help for a list of possible options
[ERROR] [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack
trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven
using the -X switch to enable full debug logging. [ERROR] [ERROR] For more
information about the errors and possible solutions, please read the following
articles: [ERROR] [Help 1] [ERROR] [ERROR] After correcting the problems, you
can resume the build with the command [ERROR] mvn <args> -rf :pinot-spi
**@jackie.jxt:** Please use java 11 to compile the project
**@syedakram93:** ok, but i cant use java 11 to run this pinot
**@syedakram93:** BTW, earlier using jdk8, it was success
**@jackie.jxt:** Yeah, we recently change the default jdk to 11. You can also
use jdk 8 to build adding this option `-Djdk.version=8`
**@syedakram93:** while running maven, shall i use `-Djdk.version=8` ?
**@jackie.jxt:** Yes. `mvn clean install ... -Djdk.version=8`
**@jackie.jxt:** More context here:
**@syedakram93:** ya working , thanks
**@syedakram93:** it would be better if you update the doc
**@jackie.jxt:** Yeah, thanks for pointing that out
**@syedakram93:** Exception while creating a build using source
**@syedakram93:**
**@syedakram93:** Apache Maven 3.6.3 Maven home: /usr/share/maven Java
version: 1.8.0_292, vendor: Private Build, runtime:
/usr/lib/jvm/java-8-openjdk-amd64/jre Default locale: en_IN, platform
encoding: UTF-8 OS name: "linux", version: "5.8.0-55-generic", arch: "amd64",
family: "unix"
**@syedakram93:** can someone help?
**@syedakram93:** stack trace:
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project pinot-spi: Compilation failure javac: invalid target release: 11
Usage: javac <options> <source files> use -help for a list of possible options
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:215) at
org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156) at
org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148) at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117) at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:81) at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:56) at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute
(DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute
(DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute
(DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute
(MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain
(MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke
(Method.java:498) at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:282) at
org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:347) Caused by:
org.apache.maven.plugin.compiler.CompilationFailureException: Compilation
failure
**@karthick.nitt:** @karthick.nitt has joined the channel
**@mark:** @mark has joined the channel
**@ajithreddy118:** @ajithreddy118 has joined the channel
### _#onboarding_
**@karthick.nitt:** @karthick.nitt has joined the channel
### _#custom-aggregators_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#docs_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#aggregators_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#pinot-dev_
**@sriramdas.sivasai:** @sriramdas.sivasai has joined the channel
**@mark:** @mark has joined the channel
**@srikanth:** @srikanth has joined the channel
**@santosh.reddy:** Can we have multiple zookeeper for pinot?
**@ssubrama:** This is more a question for zookeeper, but yes. Zookeeper by
itself support multiple servers, managing quorum. They suggest an odd number
of them.
### _#announcements_
**@mark:** @mark has joined the channel
### _#pinot-docs_
**@karthick.nitt:** @karthick.nitt has joined the channel
### _#aggregate-metrics-change_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#pinot-perf-tuning_
**@juraj.pohanka:** @juraj.pohanka has joined the channel
### _#getting-started_
**@sriramdas.sivasai:** @sriramdas.sivasai has joined the channel
**@mark:** @mark has joined the channel
**@srikanth:** @srikanth has joined the channel
### _#pinot-docsrus_
**@karthick.nitt:** @karthick.nitt has joined the channel
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org