You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2021/05/28 02:00:20 UTC
Apache Pinot Daily Email Digest (2021-05-27)
### _#pinot-helix_
**@syedakram93:** @syedakram93 has joined the channel
### _#troubleshooting_
**@syedakram93:** enabled authorization in my setup, and successfully added
schema, while trying to ingest data, i am getting below error. *bin/pinot-
admin.sh LaunchDataIngestionJob -jobSpecFile /home/sas/apache-pinot-
incubating-0.8.0-SNAPSHOT-
bin/examples/batch/airlineStats/ingestionJobSpec.yaml* Got exception to kick
off standalone data ingestion job - java.lang.RuntimeException: Failed to read
from Schema URI - '' at
org.apache.pinot.common.segment.generation.SegmentGenerationUtils.getSchema(SegmentGenerationUtils.java:87)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.init(SegmentGenerationJobRunner.java:144)
~[pinot-batch-ingestion-standalone-0.8.0-SNAPSHOT-
shaded.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:140)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:113)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:132)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17]
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL: at .(HttpURLConnection.java:1894) ~[?:1.8.0_212] at
.(HttpURLConnection.java:1492) ~[?:1.8.0_212] at
org.apache.pinot.common.segment.generation.SegmentGenerationUtils.fetchUrl(SegmentGenerationUtils.java:231)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.common.segment.generation.SegmentGenerationUtils.getSchema(SegmentGenerationUtils.java:85)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] ...
6 more Exception caught: java.lang.RuntimeException: Failed to read from
Schema URI - '' at
org.apache.pinot.common.segment.generation.SegmentGenerationUtils.getSchema(SegmentGenerationUtils.java:87)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.init(SegmentGenerationJobRunner.java:144)
~[pinot-batch-ingestion-standalone-0.8.0-SNAPSHOT-
shaded.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:140)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:113)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:132)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:164)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:184)
[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17]
Caused by: java.io.IOException: Server returned HTTP response code: 403 for
URL: at .(HttpURLConnection.java:1894) ~[?:1.8.0_212] at
.(HttpURLConnection.java:1492) ~[?:1.8.0_212] at
org.apache.pinot.common.segment.generation.SegmentGenerationUtils.fetchUrl(SegmentGenerationUtils.java:231)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] at
org.apache.pinot.common.segment.generation.SegmentGenerationUtils.getSchema(SegmentGenerationUtils.java:85)
~[pinot-all-0.8.0-SNAPSHOT-jar-with-
dependencies.jar:0.8.0-SNAPSHOT-9becc57eec981d71d5b45af5da7b720840d18f17] ...
6 more
**@syedakram93:** can someone please help?
**@nadeemsadim:** is there some way to delete few rows in pinot table ..
means i want to get rid of some garbage data consumed into pinot .. so can i
run some delete query to delete only rows matching that query
**@mayanks:** Deletion is at segment level, not row level
**@nadeemsadim:** ok
**@jmeyer:** Hello :smile: Does Pinot have any support for hierarchical
aggregations ? Say we have a tree like structure, with values at the leaves,
is there an efficient way to get the values at the intermediates & root nodes
? (Tree size can be ~10 layer deep, with ~10000-100000 leaves)
**@mayanks:** Have you looked at Star tree index?
**@jmeyer:** I've read a bit about it, not sure it could fit that use case ?
(looks like it could :slightly_smiling_face:)
**@mayanks:**
**@jmeyer:** Do you have pointers on how to configure it / how the data should
look like to make it work in that scenario ? Maybe an important detail is that
I should have said "_tree*S"*_ Ideally only known at runtime (not integration
time) I'll look into it first
**@shaileshjha061:** Hi @mayanks @dlavoie Something wrong with GCS
Integration. Pinot server Pods went into crashloopbackoff when I try to
integrate with GCS Error Log: ```ERROR [PinotFSFactory] [Start a Pinot
[SERVER]] Could not instantiate file system for class
org.apache.pinot.plugin.filesystem.GcsPinotFS with scheme gs```
@mohamed.sultan
**@dlavoie:** A full stacktrace from the logs always help to troubleshoot.
**@shaileshjha061:** let me send
**@mohamed.sultan:** found the issue @dlavoie
**@mayanks:** @mohamed.sultan Could you please share the issue? Also, we are
request help from community to help improve our docs. Would you be intrested
in documenting what you found? If so, please join <#C023BNDT0N8|pinot-docsrus>
**@jmeyer:** Does any sort of query cache exist in Pinot ?
**@mayanks:** No, there isn't one right now. For realtime cases, the data is
changing in realtime, so caching the query result might not make sense.
**@jmeyer:** Yes in the case of realtime table the case would be invalidated
very often In case of offline tables, I guess the best way is to add an
external cache at the REST API layer ? And invalidate it whenever new segments
are pushed
**@mayanks:** Added to FAQ:
**@jmeyer:** Looks great and clear, thanks @mayanks!
**@machhindra.nale:** Trying to browse Pinot in Superset. Following query
works fine in the Pinot Query console. BUT fails in Superset. Notice the
special character in the metric. Is that something because of Pinot SQLAlchemy
driver? ```SELECT DATETIMECONVERT(metricTime, '1:MILLISECONDS:EPOCH',
'1:MILLISECONDS:EPOCH', '1:MINUTES'), AVG(metricValue) AS "AVG_1" FROM
metric_v6.metric_v6 WHERE metricTime >= 1621555200000 AND metricTime <
1622160000000 AND metric = 'CECCP%' GROUP BY DATETIMECONVERT(metricTime,
'1:MILLISECONDS:EPOCH', '1:MILLISECONDS:EPOCH', '1:MINUTES') LIMIT 10000```
Screenshot of the Superset SQL Editor -
**@mayanks:** Can you check broker log to see what query was sent to Pinot?
**@fx19880617:** Is it the % ?
**@fx19880617:** It could be the python client translation
### _#pinot-docs_
**@syedakram93:** @syedakram93 has joined the channel
### _#presto-pinot-connector_
**@syedakram93:** @syedakram93 has joined the channel
### _#minion-improvements_
**@laxman:** Have a basic tuning related question. Whats the recommended size
for REALTIME and OFFLINE tables?
**@g.kishore:** Segment size or table size
**@laxman:** segment size
**@g.kishore:** 100 to 500mb
**@laxman:** I had gone through this documentation
**@laxman:**
**@laxman:** 100 to 500 MB for REALTIME right. How about OFFLINE segments?
**@laxman:** is it same for both?
**@npawar:** yes, same for both
**@laxman:** okay. thank you
### _#pinot-docsrus_
**@patidar.rahul8392:** @patidar.rahul8392 has joined the channel
**@syedakram93:** @syedakram93 has joined the channel
**@jmeyer:** @jmeyer has joined the channel
**@mayanks:** Thanks for joining folks. I recommend helping out questions
being asked by checking if it could have been answered by doc or faq.
**@mayanks:** Don't by shy folks :grin:, let's get the party started. @jmeyer
You asked a question where star tree could be the answer. Perhaps that could
go into the FaQ?
**@jmeyer:** Haha I'm still trying to figure out how it would fit, but if I
find a way to use star tree in my usecase, maybe we could add a similar
example in the docs ? (as mine is quite different from the current one)
**@mayanks:** No worries.
**@jmeyer:** Anyway, this channel is a good move I think, I'll suggest
improvements whenever I see some :slightly_smiling_face:
**@mayanks:** Here's a simple one for anyone to take. I already answered, but
we can add it to FaQ:
**@jmeyer:** Yep, maybe (if not already present), have an introduction about
the limitations & differences between OLTP & OLAP (in the context of Pinot of
course) ?
**@mayanks:** Sure, go for it, file a PR.
**@mayanks:** I have added a link on how to contribute to the docs in the
channel topic
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org