You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pinot.apache.org by Pinot Slack Email Digest <sn...@apache.org> on 2021/03/25 02:00:18 UTC
Apache Pinot Daily Email Digest (2021-03-24)
### _#general_
**@punit.kj:** @punit.kj has joined the channel
**@chris.lu:** @chris.lu has joined the channel
**@karinwolok1:** :tada: Today! Catch @fx19880617 and @g.kishore’s
presentations & panel talks at PrestoCon! (conference is FREE!)
**@kvndesign.mail:** @kvndesign.mail has joined the channel
### _#random_
**@punit.kj:** @punit.kj has joined the channel
**@chris.lu:** @chris.lu has joined the channel
**@kvndesign.mail:** @kvndesign.mail has joined the channel
### _#feat-presto-connector_
**@punit.kj:** @punit.kj has joined the channel
### _#troubleshooting_
**@punit.kj:** @punit.kj has joined the channel
**@phuchdh:** Hi guys, i need to set group.id in LLC, but i cannot find any
configuration for that. In my company, we need permission for specify group.id
to consume message. ```2021/03/24 06:13:18.879 WARN [ConsumerCoordinator]
[RuleLogsQC__2__1__20210319T0728Z] [Consumer clientId=consumer-12735,
groupId=] Synchronous auto-commit of offsets {c1.elk.db-gamification-consumer-
log.qc-2=OffsetAndMetadata{offset=17073, metadata=''}} failed: Not authorized
to access group:```
**@chris.lu:** @chris.lu has joined the channel
**@ravi.maddi:** Hi All,
**@ravi.maddi:** Hi All, I am using JSON Indexing, I have a scenario, In my
persons json content, a column 'addresses' contains a list of addresses. Like
this: Need Help :slightly_smiling_face: ``` { "personId": "9878", "addresses":
[ { "doorNum": "45456", "Street": "Washington Road", "area": "sector-1" }, {
"doorNum": "676756", "Street": "Washington Road", "area": "sector-2" }, {
"doorNum": "768768", "Street": "Washington Road", "area": "sector-4" } ] } {
"personId": "68768", "addresses": [ { "doorNum": "45456", "Street":
"Washington Road", "area": "sector-1" }, { "doorNum": "676756", "Street":
"Washington Road", "area": "sector-2" }, { "doorNum": "768768", "Street":
"Washington Road", "area": "sector-4" } ] }``` In Schema config file I
mentioned like this: ``` { "name": "addresses", "dataType": "STRING",
"maxLength": 2147483647, "singleValueField": false },``` In Table Config file
I mentioned like this: ``` "jsonIndexColumns": [ "addresses" ],``` But I am
not able to find data in Pinot Query Console, I am not able to find any error
in any log. Need Help
**@tanmay.movva:** Hello, We have setup a monitoring dashboard on grafana for
pinot. We are currently using the image with `latest` tag and we observe that
few metrics related to query response time have been coming as `0` . We didn’t
change any config on pinot or the grafana query, so wanted to know if there
was any breaking change merged recently? Here is a screen shot of the
dashboard.
**@tanmay.movva:** Affected Metrics - ```pinot_broker_queryExecution_*
pinot_server_freshnessLagMs_* pinot_server_totalQueryTime_*
pinot_server_queryProcessing_*```
**@g.kishore:** Can you check the JMX metric names. There was a change in the
metric name
**@g.kishore:** @fx19880617
**@fx19880617:** I will check, could be related to the recent metrics plugin
extraction @jlli just fyi
**@fx19880617:** @tanmay.movva do you know which image you are using
previously? Or when you restarted your deployment before this restart ?
**@tanmay.movva:** We are/were using the image with `latest` tag only. From
the metrics dashboard, we are not able to see the metrics after 17th March(We
restarted on that day).
**@fx19880617:** I will check on that
**@tanmay.movva:** > Can you check the JMX metric names. There was a change in
the metric name Checked for alternate metrics for those but didn’t find any.
**@fx19880617:** did you see anything when you run `curl localhost:8080/jmx`
in the container?
**@fx19880617:** also are the controller metrics there?
**@tanmay.movva:** > also are the controller metrics there? Yes.
**@fx19880617:** hmm, controller are also restarted?
**@tanmay.movva:** They were restarted, but not sure if the image was pulled
for them. Will force a image pull for all the services once.
**@jlli:** I don’t think it’s related to metrics plugin extraction. The latest
official release was cut based on the commit hash on 02/24 while metric plugin
extraction was merged 03/08
**@tanmay.movva:** > hmm, controller are also restarted? Just did this, don’t
think controller metrics are affected. I randomly checked few metrics and they
were coming in. Any specific metrics to check for?
**@tanmay.movva:** > he latest official release was cut based on the commit
hash on 02/24 while metric plugin extraction was merged 03/08 We are not using
the officially released tag yet. We are using the `latest` tag for pinot
image. Is `0.7.0` tag released?
**@jlli:** So the table query latency should be emitted from pinot-broker
right? I think you should take a look at the health of broker metric
**@tanmay.movva:** I can see the QPS metrics from broker but not query latency
metrics.
**@tanmay.movva:** And helix_connected_value, Status of broker on UI all are
OK. Queries are also being processed.
**@fx19880617:** can u list the metrics missing, then I can check
**@tanmay.movva:** Hmm..Will try to get list of all metrics, but from the
panels I setup in the dashboard, these are the metrics missing currently.
**@fx19880617:** hmm, I tried latest image and found that those metrics are
there: ```root@pinot-server-0:/opt/pinot# curl localhost:8080/jmx|grep
freshnessLagMs pinot_server_freshnessLagMs_MeanRate Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>MeanRate) #
TYPE pinot_server_freshnessLagMs_MeanRate untyped
pinot_server_freshnessLagMs_MeanRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_MeanRate{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_OneMinuteRate Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>OneMinuteRate)
# TYPE pinot_server_freshnessLagMs_OneMinuteRate untyped
pinot_server_freshnessLagMs_OneMinuteRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_OneMinuteRate{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_StdDev Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>StdDev) # TYPE
pinot_server_freshnessLagMs_StdDev untyped
pinot_server_freshnessLagMs_StdDev{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_StdDev{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_95thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>95thPercentile)
# TYPE pinot_server_freshnessLagMs_95thPercentile untyped
pinot_server_freshnessLagMs_95thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_95thPercentile{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_99thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>99thPercentile)
# TYPE pinot_server_freshnessLagMs_99thPercentile untyped
pinot_server_freshnessLagMs_99thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_99thPercentile{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_Mean Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Mean) # TYPE
pinot_server_freshnessLagMs_Mean untyped
pinot_server_freshnessLagMs_Mean{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_Mean{table="airlineStats",tableType="REALTIME",}
0.0 100 180k 100 180k 0 0 204k # HELP
pinot_server_freshnessLagMs_FifteenMinuteRate Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>FifteenMinuteRate)
# TYPE pinot_server_freshnessLagMs_FifteenMinuteRate untyped 0
pinot_server_freshnessLagMs_FifteenMinuteRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
-pinot_server_freshnessLagMs_FifteenMinuteRate{table="airlineStats",tableType="REALTIME",}
0.0 -# HELP pinot_server_freshnessLagMs_Min Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Min) :-# TYPE
pinot_server_freshnessLagMs_Min untyped
-:pinot_server_freshnessLagMs_Min{table="airlineStatsAvro",tableType="REALTIME",}
0.0
-pinot_server_freshnessLagMs_Min{table="airlineStats",tableType="REALTIME",}
0.0 \- -# HELP pinot_server_freshnessLagMs_Count Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Count) -:--:--
--:# TYPE pinot_server_freshnessLagMs_Count untyped
\--pinot_server_freshnessLagMs_Count{table="airlineStatsAvro",tableType="REALTIME",}
0.0
:pinot_server_freshnessLagMs_Count{table="airlineStats",tableType="REALTIME",}
0.0 \-- # HELP pinot_server_freshnessLagMs_999thPercentile Attribute exposed
for management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>999thPercentile)
2# TYPE pinot_server_freshnessLagMs_999thPercentile untyped
04pinot_server_freshnessLagMs_999thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0 k
pinot_server_freshnessLagMs_999thPercentile{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_50thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>50thPercentile)
# TYPE pinot_server_freshnessLagMs_50thPercentile untyped
pinot_server_freshnessLagMs_50thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_50thPercentile{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_75thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>75thPercentile)
# TYPE pinot_server_freshnessLagMs_75thPercentile untyped
pinot_server_freshnessLagMs_75thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_75thPercentile{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_FiveMinuteRate Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>FiveMinuteRate)
# TYPE pinot_server_freshnessLagMs_FiveMinuteRate untyped
pinot_server_freshnessLagMs_FiveMinuteRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_FiveMinuteRate{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_98thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>98thPercentile)
# TYPE pinot_server_freshnessLagMs_98thPercentile untyped
pinot_server_freshnessLagMs_98thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_98thPercentile{table="airlineStats",tableType="REALTIME",}
0.0 # HELP pinot_server_freshnessLagMs_Max Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.airlineStatsAvro_REALTIME.freshnessLagMs"><>Max) # TYPE
pinot_server_freshnessLagMs_Max untyped
pinot_server_freshnessLagMs_Max{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_freshnessLagMs_Max{table="airlineStats",tableType="REALTIME",}
0.0```
**@fx19880617:** ```root@pinot-server-0:/opt/pinot# curl
localhost:8080/jmx|grep totalQueryTime pinot_server_totalQueryTime_Count
Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Count) # TYPE
pinot_server_totalQueryTime_Count untyped
pinot_server_totalQueryTime_Count{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Count{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Count{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Count{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_Count{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_Count{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_Count{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_FifteenMinuteRate Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>FifteenMinuteRate)
# TYPE pinot_server_totalQueryTime_FifteenMinuteRate untyped
pinot_server_totalQueryTime_FifteenMinuteRate{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_FifteenMinuteRate{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_Max Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Max) # TYPE
pinot_server_totalQueryTime_Max untyped
pinot_server_totalQueryTime_Max{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Max{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Max{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Max{table="airlineStatsAvro",tableType="REALTIME",}
0.0 pinot_server_totalQueryTime_Max{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_Max{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_Max{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_50thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>50thPercentile) #
TYPE pinot_server_totalQueryTime_50thPercentile untyped
pinot_server_totalQueryTime_50thPercentile{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_50thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_50thPercentile{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_50thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_50thPercentile{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_50thPercentile{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_50thPercentile{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_98thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>98thPercentile) #
TYPE pinot_server_totalQueryTime_98thPercentile untyped
pinot_server_totalQueryTime_98thPercentile{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_98thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_98thPercentile{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_98thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_98thPercentile{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_98thPercentile{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_98thPercentile{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_75thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>75thPercentile) #
TYPE pinot_server_totalQueryTime_75thPercentile untyped
pinot_server_totalQueryTime_75thPercentile{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_75thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_75thPercentile{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_75thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_75thPercentile{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_75thPercentile{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_75thPercentile{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_FiveMinuteRate Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>FiveMinuteRate) #
TYPE pinot_server_totalQueryTime_FiveMinuteRate untyped
pinot_server_totalQueryTime_FiveMinuteRate{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_FiveMinuteRate{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_StdDev Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>StdDev) # TYPE
pinot_server_totalQueryTime_StdDev untyped
pinot_server_totalQueryTime_StdDev{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_StdDev{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_StdDev{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_StdDev{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_StdDev{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_StdDev{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_StdDev{table="complexWebsite",tableType="OFFLINE",}
0.0 100 180k 100 180k 0 0 204k 0 --:--:-- --:--:-- --:--:-- 204k # HELP
pinot_server_totalQueryTime_OneMinuteRate Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>OneMinuteRate) #
TYPE pinot_server_totalQueryTime_OneMinuteRate untyped
pinot_server_totalQueryTime_OneMinuteRate{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_OneMinuteRate{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_OneMinuteRate{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_OneMinuteRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_OneMinuteRate{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_OneMinuteRate{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_OneMinuteRate{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_Min Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Min) # TYPE
pinot_server_totalQueryTime_Min untyped
pinot_server_totalQueryTime_Min{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Min{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Min{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Min{table="airlineStatsAvro",tableType="REALTIME",}
0.0 pinot_server_totalQueryTime_Min{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_Min{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_Min{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_Mean Attribute exposed for management
("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>Mean) # TYPE
pinot_server_totalQueryTime_Mean untyped
pinot_server_totalQueryTime_Mean{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Mean{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Mean{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_Mean{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_Mean{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_Mean{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_Mean{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_MeanRate Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>MeanRate) # TYPE
pinot_server_totalQueryTime_MeanRate untyped
pinot_server_totalQueryTime_MeanRate{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_MeanRate{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_MeanRate{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_MeanRate{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_MeanRate{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_MeanRate{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_MeanRate{table="complexWebsite",tableType="OFFLINE",}
0.0 # HELP pinot_server_totalQueryTime_95thPercentile Attribute exposed for
management ("org.apache.pinot.common.metrics"<type="ServerMetrics",
name="pinot.server.simpleWebsite_OFFLINE.totalQueryTime"><>95thPercentile) #
TYPE pinot_server_totalQueryTime_95thPercentile untyped
pinot_server_totalQueryTime_95thPercentile{table="simpleWebsite",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_95thPercentile{table="covid19_confirmed_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_95thPercentile{table="covid19_deaths_us",tableType="OFFLINE",}
0.0
pinot_server_totalQueryTime_95thPercentile{table="airlineStatsAvro",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_95thPercentile{table="covid19_confirmed",tableType="us",}
0.0
pinot_server_totalQueryTime_95thPercentile{table="airlineStats",tableType="REALTIME",}
0.0
pinot_server_totalQueryTime_95thPercentile{table="complexWebsite",tableType="OFFLINE",}
0.0```
**@fx19880617:** can you check if the jmx server are reporting those metrics?
**@fx19880617:** also check if your prometheus metrics exporter are deployed
**@tanmay.movva:** Those metrics are coming in. But they are coming in as `0`
. They aren’t being updated. Even the metrics you shared have zero value. But
when queries are running then execution Time shouldn’t be zero no?
**@tanmay.movva:** My problem was not that they were missing, but the value of
those metrics is always zero. Which wasn’t the case earlier.
**@fx19880617:** i see, i will double check
**@chxing:** Hi All, I got an error Caused by: .BindException: Address
already in use at .bind0(Native Method) at .bind(Net.java:433) at
.bind(Net.java:425) at
.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220) at
.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85) at
.httpserver.ServerImpl.bind(ServerImpl.java:133) at
.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54) at
.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145) at
.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31) ... 6 more FATAL ERROR in
native method: processing of -javaagent failed When I deploying controller,
broker, server with -javaagent:jmx_prometheus_javaagent-0.12.0.jar=9066 in one
vm server, Do you know how to send metrics to prometheus:9066 in one node
**@dlavoie:** the prometheus exporter is not a client. It’s a server. so when
you define `jmx_prometheus_javaagent-0.12.0.jar=9066`, it opens a port 9066 on
your Pinot process for Prometheus to scrape metrics.
**@dlavoie:** This configuration is not for your prometheus server port. You
need a distinct port. and configure a prometheus scrape config that reference
this pinot host and port.
**@chxing:** Thx, I can config different ports in prometheus to collect those
metrics
**@dlavoie:** Yes
**@dlavoie:** It,s all part of your prometheus scrape config
**@dlavoie:** If everything is running on one node, you will need distinct
port on the javaagent config for each pinot component
**@chxing:** yes I just changed <
**@prshnt.1314:** Hi all, I'm trying to connect presto to pinot cluster using
connector. But I'm facing this null pointer exception. Has anyone else faced
this or give some hints to solve this.
**@joshhighley:** When ingesting json data, do the json objects have to be
flat? I can't find docs about parsing json object trees, but I see the github
event example is a tree
**@mayanks:** It depends on what you are trying to do: ```1. If you want to
store data as JSON blob in Pinot and use JSON indexing: 2\. If you want to
extract data from nested input JSON and store in a flat schema: ```
**@joshhighley:** #2. The row data is several levels deep, and it's not an
array but rather a list of id properties { node1: { node2: { node3: { id1: {
col1: value1, col2: value2 }, id2: { col1: value1, col2: value2 } } } }
**@mayanks:** You can write a transform to extract that data and map to a
column in Pinot schema
**@mayanks:** List can map to multi-valued columns
**@joshhighley:** sorry, hit send before my example was finished. id1 is a
row, id2 is a row, etc
**@mayanks:** You should still be able to write per column transform (would be
a little inefficient to do it once per column)?
**@joshhighley:** How does Pinot decide what the row level object is? ex: id1,
id2, etc vs some other object
**@mayanks:** Your transform function would need to decide that
**@kvndesign.mail:** @kvndesign.mail has joined the channel
**@joshhighley:** The server hosting our Pinot got rebooted and when we
restarted the components, we have no segments, no tables and no schemas. I see
in the server logs that it's looking in the correct directories for instance
data and instance segment, and I've confirmed there's data in those
directories. Pinot server isn't loading anything though?
**@mayanks:** What do you see in the ideal-state? Does it claim that the
server should host data?
**@joshhighley:** where do I check that?
**@mayanks:** In the cluster manager or in zookeeper (via query console)
**@mayanks:** Ideal-state reflects which segments are "expected" to be on
which servers.
**@mayanks:** External-view reflects the current state of the system.
**@mayanks:** Let's first see what Ideal-State says. If it claims that the
server should host segments, then we check external view.
**@joshhighley:** those only have 'brokerResource' and
'leadControllerResource'
**@mayanks:** Hmm, nothing for the table?
**@joshhighley:** under idealStates? no
**@mayanks:** Does cluster manager show any tables tab?
**@mayanks:** or the rest-api `<controllerHost>:<controllerPort>/tables`?
**@mayanks:** If none of them show the table, then there is no table
:grimacing:
**@joshhighley:** ugh. pinot-admin.sh doesn't log correctly so we don't even
have log files
**@mayanks:** I think it should log correctly, unless you have overwritten
log4j configs?
**@mayanks:** If you still feel it is not doing so, please file an issue and
we will address it.
**@jackie.jxt:** If only these 2 records exist in ideal state, you need to
check if zookeeper somehow gets all the contents deleted during the reboot
### _#pinot-dev_
**@punit.kj:** @punit.kj has joined the channel
### _#pinot-startup_
**@ravi.maddi:** Hi All, I am using JSON Indexing, I have a scenario, In my
persons json content, a column 'addresses' contains a list of addresses. Like
this: Need Help :slightly_smiling_face: ``` { "personId": "9878", "addresses":
[ { "doorNum": "45456", "Street": "Washington Road", "area": "sector-1" }, {
"doorNum": "676756", "Street": "Washington Road", "area": "sector-2" }, {
"doorNum": "768768", "Street": "Washington Road", "area": "sector-4" } ] } {
"personId": "68768", "addresses": [ { "doorNum": "45456", "Street":
"Washington Road", "area": "sector-1" }, { "doorNum": "676756", "Street":
"Washington Road", "area": "sector-2" }, { "doorNum": "768768", "Street":
"Washington Road", "area": "sector-4" } ] }``` In Schema config file I
mentioned like this: ``` { "name": "addresses", "dataType": "STRING",
"maxLength": 2147483647, "singleValueField": false },``` In Table Config file
I mentioned like this: ``` "jsonIndexColumns": [ "addresses" ],``` But I am
not able to find data in Pinot Query Console, I am not able to find any error
in any log. Need Help @mayanks Can you help me.
**@mayanks:** Yes will look at in shortly
**@mayanks:** What about select * ? Does it return anything
**@mayanks:** If not what is the error you get
**@ravi.maddi:** select * giving zero records.
**@mayanks:** no errors?
**@ravi.maddi:** no errors
**@mayanks:** also can you check external view from console to see if there
are any segments?
**@mayanks:** We want to check if this is an issue because segment is not
loaded in Pinot, or if the segment is not generated correctly
**@mayanks:** If you have access to the segment directory, can you post the
metadata.properties file inside of the segment dir?
**@ravi.maddi:** Ok, I will check
**@ravi.maddi:** I am using quick start streaming to start my cluster.
**@mayanks:** can we do a zoom call now?
**@ravi.maddi:** I am not able to find any file with name metadata.properties
**@mayanks:** easier to debug that way
**@ravi.maddi:** Yes Mayank
**@mayanks:** let me send a link
**@ravi.maddi:** ok
**@mayanks:** ```Mayank Shrivastava is inviting you to a scheduled Zoom
meeting. Topic: Mayank Shrivastava's Personal Meeting Room Join Zoom Meeting
Meeting ID: 764 130 6006 Passcode: wP6Gng```
**@mayanks:** ok i have logged in ^^
**@mayanks:** are you joining @ravi.maddi?
**@ravi.maddi:** joined
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org