You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <ap...@gmail.com> on 2021/08/25 02:00:22 UTC
Apache Pinot Daily Email Digest (2021-08-24)
### _#general_
**@kavya8428:** @kavya8428 has joined the channel
**@sulthon.poetra.ahmad:** @sulthon.poetra.ahmad has joined the channel
**@karinwolok1:** Today! with presenters @steotia and @guo.980 :heart_eyes:
**@snlee:** Hello Community, We are pleased to announce that Apache Pinot
0.8.0 is released! Apache Pinot is a distributed columnar storage engine that
can ingest data in real-time and serve analytical queries at low latency. The
release can be downloaded at The release note is available at Additional
resources - Project website: Getting started: Pinot developer blogs: Intro
to Pinot Video: Join Pinot Community - Twitter: Meetup: Slack channel:
Best Regards, Apache Pinot Team
**@karinwolok1:** If there's any .NET + Pinot devs out there that want to
share their story - .Net conf has it's open CFP and accepting talk
submissions! :slightly_smiling_face:
**@karinwolok1:** For anyone who missed the meetup today
**@fritzb:** @snlee where can we find the docker image for release-0.8.0 ? I
can't seem to pull `apachepinot/pinot:release-0.8.0`
**@xiangfu0:** I will do it
**@jai.patel856:** what are the mechanisms to throttle the speed at which
messages are ingested via realtime?
### _#random_
**@kavya8428:** @kavya8428 has joined the channel
**@sulthon.poetra.ahmad:** @sulthon.poetra.ahmad has joined the channel
### _#troubleshooting_
**@kangren.chia:** so far from what i’ve seen there are some vague references
to default cpu/memory sizing on the docs: have you seen this @nadeemsadim?
**@kavya8428:** @kavya8428 has joined the channel
**@lovenishgoyal:** Hi All, Can anyone help me to connect tableau to pinot ?
we are getting below error
**@xiangfu0:** which version are you running ? have you tried 0.8.0?
**@xiangfu0:** @kharekartik fyi
**@lovenishgoyal:** pinot is running on 0.7.0
**@lovenishgoyal:** for pinot-jdbc we tried 0.7.0 and 0.8.0 both
**@xiangfu0:** hmm i think it’s missing http client lib
**@xiangfu0:** can you try to download and put it
**@xiangfu0:**
**@lovenishgoyal:** Sure, let me try this
**@sulthon.poetra.ahmad:** @sulthon.poetra.ahmad has joined the channel
**@syedakram93:** WARNING: All illegal access operations will be denied in a
future release Exception in thread "main" java.lang.NoSuchMethodError:
'org.apache.htrace.HTraceConfiguration
org.apache.hadoop.tracing.TraceUtils.wrapHadoopConf(org.apache.hadoop.conf.Configuration)'
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:635) at
org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619) at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2812) at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100) at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849) at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:181) at
org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:65)
at
org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52)
at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72)
at
org.apache.pinot.controller.BaseControllerStarter.initPinotFSFactory(BaseControllerStarter.java:518)
at
org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:358)
at
org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:308)
at
org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:123)
at
org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:93)
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:233)
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:285)
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:232)
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:182)
at
org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:149)
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166)
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186)
at org.apache.pinot.tools.admin.PinotController.main(PinotController.java:35)
**@syedakram93:** iam getting above exception while starting up controller
using hdfs as deepstore
**@syedakram93:** export HADOOP_HOME=/home/sas/zohohadoop/hdfs export
HADOOP_VERSION=2.8.5 export HADOOP_GUAVA_VERSION=11.0.2 export
HADOOP_GSON_VERSION=2.2.4 #export GC_LOG_LOCATION=/path/to/gc/log/file export
PINOT_VERSION=0.8.0 export PINOT_DISTRIBUTION_DIR=/home/sas/pinot/ export
SERVER_CONF_DIR=/home/sas/pinot/conf/ export ZOOKEEPER_ADDRESS=localhost:2191
export CLASSPATH_PREFIX="${HADOOP_HOME}/lib/hadoop-
hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/hadoop-
annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/hadoop-
auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/hadoop-
common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/lib/gson-${HADOOP_GSON_VERSION}.jar:${HADOOP_HOME}/lib/htrace-
core4-4.0.1-incubating.jar" export JAVA_OPTS="-Xms4G -Xmx4G"
${PINOT_DISTRIBUTION_DIR}/bin/start-controller.sh -configFileName
${SERVER_CONF_DIR}/controller.conf
**@syedakram93:** using above config
**@syedakram93:** can someone help?
**@syedakram93:**
**@xiangfu0:** try out a lower hadoop version? the doc is using 2.7.1
**@xiangfu0:** seems there is method signature change in 2.8.x
**@syedakram93:** @xiangfu0 @npawar @mayanks
**@will.gan:** Hi, if I want to have a time column that's essentially another
column (say epoch_minutes) but bucketed (say every 5 minutes), my
understanding was that I can create a column with format `1:MINUTES:EPOCH` and
granularity `5:MINUTES` , and that Pinot would handle it for me. I don't think
that's the case though, i.e. I have to write an ingestion transform right?
**@npawar:** you have to write transform function
**@will.gan:** ok thanks, but out of curiosity if that's the case couldn't I
just make my format `5:MINUTES:EPOCH` ? How does granularity differ?
**@npawar:** `5:MINUTES:EPOCH` would mean, `epoch_millis/(1000*60*5)` whereas,
`1:MINUTES:EPOCH` with granularity `5:MINUTES` would mean,
`epoch_millis/(1000*60)`, with value rounded to the nearest 5 minutes
**@npawar:** the former is not a very natural way of representing. The latter
is more commonly used
**@will.gan:** okay
### _#pinot-dev_
**@luisfernandez:** @luisfernandez has joined the channel
**@s.swaminathan:** @s.swaminathan has joined the channel
**@art.artik:** @art.artik has joined the channel
### _#getting-started_
**@s.swaminathan:** @s.swaminathan has joined the channel
**@art.artik:** @art.artik has joined the channel
**@tiger:** Is there a way to enable/disable individual segments?
**@kulbir.nijjer:** @tiger can you elaborate on what’s the use case ?
**@tiger:** I want to batch ingest data on a per day basis, but only enable
the most recent day once it has finished ingesting. Each day would end up
consisting of many segments. Is there a suggested method of doing this?
**@kulbir.nijjer:** Hmm there is no such API to enable/disable individual
segments, only deleting a segment or multiple(JSON array) if you have a list.
Just curious why table only needs to have data for most recent day and not
others ? If older days data is not needed ever then simply configure the
`retention*` settings in the table appropriately: cc @mayanks
**@mayanks:** @tiger the recommended way of doing so is to have time filter in
the query to explicitly select the time range. You can additional set
retention to auto purge/delete older segments.
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org