Apache Pinot Daily Email Digest (2022-04-08)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@bhat.shruti:** @bhat.shruti has joined the channel  
 **@ysuo:** Hi, is it supported to use JSONEXTRACTSCALAR function in the
transformConfigs?  
**@kharekartik:** Hi No that is not supported but you can use `jsonPath`
function which does a similar thing. If you require a specific result data
type you can use its variations such as `jsonPathLong`, `jsonPathDouble` etc.  
**@ysuo:** @kharekartik Thanks. Another question, does jsonPath supports
regular expression?  
**@kharekartik:** Not sure. You can test out here -  we use jayway jsonpath in
our code.  
**@ysuo:** Got it. Thanks, Kartik.  
 **@satyam.raj:** Hey guys, Is there a way to connect Pinot to a secure Kafka
cluster using Jaas configuration?  
**@mayanks:** @slack1 might have some insights here.  
**@slack1:** Hi Satyam, yes. I remember @patidar.rahul8392 was settings up
something similar some time ago. If you look up his posts via search you’ll
find a number of actual configs  
 **@francois:** Hi getting deeper in my  process dev many thanks to the pinot
team for the awesome support and code quality :wink: . Is there any way to
upload a realtime segment ? Like purging it and refreshing it ? I do not
really inderstand why this is not possible if the segment is not consuming
:confused: And I’m a bit surprised of the API getting an OFFLINE table for the
uploadProcess instead my original type REALTIME  
**@mark.needham:** at the moment you can't replace a real-time segment, but I
think someone might be working on it. @mayanks will have more info.  
**@francois:** I can see it’s replacing my REALTIME to OFFLINE for the table
name but don’t know where :confused:  
**@mark.needham:** at the moment you can only replace offline segments, real-
time ones won't work  
**@mark.needham:** not sure why it'd be renaming your table  
**@mark.needham:** i thought it threw an error indicating that you can't
replace real-time segments  
**@francois:** Not really it complains about segment non existing (Still in
realTime Table )`org.apache.pinot.common.exception.HttpErrorStatusException:
Got error status code: 410 (Gone) with reason: "Cannot refresh non-existing
segment, aborted uploading segment: actions__0__0__20220405T0951Z of table:
actions_OFFLINE" while sending request: `  
**@francois:** ok I will switch my logic to get the segment in OFFLINE table
an then purge them :wink:  
**@mark.needham:** There's a simple example showing one way to do that here
-->  
**@francois:** Yes already use it :wink:  
**@mark.needham:** ah cool  
 **@mrpringle:** Any plans for pinot to support
'RealtimeToOfflineSegementsTask' for upsert enabled tables. Use case is this:
1\. Normal queries; key contains a business date; only want to see the last
value for a day (as per current upsert config works well) 2\. Detail / as of
queries; want to see record up to a given timestamp for a given business date
(skipUpsert and LASTWITHTIME( col, asOfTime ). e.g. compare 11am yesterday v
today 3\. Historical want to move data beyond a week to OFFLINE 4\. Historical
want to drop data points e.g. milli seconds -> hours per day. Like reducing
data resolution over time through some data cleaning job.  
**@luisfernandez:** number 3 and 4 i believe you can do with the current
RTToOffline job  
 **@datanarch01:** @datanarch01 has joined the channel  
 **@ciro.zar:** @ciro.zar has joined the channel  
 **@weixiang.sun:** It seems that we have health API for controller. Do we
have health API for table and broker?  
**@mayanks:** There should be one for broker. Swagger should list all the
apis, you can check if there’s one for table there.  
 **@evan.galpin:** hey folks, does anyone have experience with “zombie”
tables? I attempted to add a tableSpec but was missing a field in the config.
This resulted in a null pointer exception when Pinot processed the config. As
a result, the table doesn’t exist in the UI or by querying the list API, but I
can’t create a table with the same name due to receiving an error that “table
already exists”. Presumably the spec reached ZK or something, but no where
else? What can be done to remove the zombie table?  
**@mayanks:** Hmm this seems like a bug to fix. Could you please file a GH
issue?  
**@bagi.priyank:** i ended up in a similar situation earlier this week. pinot
cluster couldn't route to kafka cluster so when i submitted table config to
create the table, the table was never created externally or visible via UI.
infact swagger api to get all tables didn't return it in the result. however
if i tried creating the table again, cluster would respond back with an error
saying that the table already exists.  
**@npawar:** I'm assuming you already tried to invoke DELETE /tables/tablename
Api but that didn't work? I've had this happen once before, but invoking a
delete cleared out the zombie states  
**@bagi.priyank:** Yep delete didn't work.  

###  _#random_

  
 **@bhat.shruti:** @bhat.shruti has joined the channel  
 **@datanarch01:** @datanarch01 has joined the channel  
 **@ciro.zar:** @ciro.zar has joined the channel  

###  _#troubleshooting_

  
 **@bhat.shruti:** @bhat.shruti has joined the channel  
 **@datanarch01:** @datanarch01 has joined the channel  
 **@ciro.zar:** @ciro.zar has joined the channel  
 **@facundo.bianco:** Hi All, I'm trying to use Pinot with AWS S3 integration
but I got this error message (whole error message will be attached):
```2022/04/08 19:30:00.576 ERROR [StartServiceManagerCommand] [main] Failed to
start a Pinot [CONTROLLER] at 35.248 since launch java.lang.RuntimeException:
Caught exception while initializing ControllerFilePathProvider at
org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:526)
~[pinot-all-0.8.0-jar-with-
dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808] ... Caused
by:
org.apache.pinot.controller.api.resources.InvalidControllerConfigException:
Caught exception while initializing file upload path provider at
org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107)
~[pinot-all-0.8.0-jar-with-
dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808] ... 11 more
Caused by: java.lang.IllegalStateException: PinotFS for scheme: s3 has not
been initialized at
shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518)
~[pinot-all-0.8.0-jar-with-
dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808] ... 11
more``` I configured Pinot's Kubernetes as ``` controller: data: dir:  #
values under "pinot.controller" at  storage: factory: s3: region: us-east-1
class: s3: org.apache.pinot.plugin.filesystem.S3PinotFS segment: fetcher: s3:
class: org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher protocols:
file,http,s3 server: segment: fetcher: protocols: file,http,s3 s3: class:
org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher instance: dataDir:
/tmp/pinot-tmp/server/index segmentTarDir: /tmp/pinot-tmp/server/segmentTars
storage: factory: class: s3: org.apache.pinot.plugin.filesystem.S3PinotFS s3:
region: us-east-1``` Pinot version is 0.8.0. Do you know where is the issue?
Thanks in advance!  
 **@mayanks:** The issue seems like PinotFs has not been initialized. Also,
any reason to not use the latest release 0.10.0?  
 **@luisfernandez:** anyone familiar with this error while executing the
QuickStart from IDE ```2022/04/08 15:53:37.705 INFO [PostQueryCommand] [main]
Executing command: PostQuery -brokerProtocol http -brokerHost 0.0.26.108
-brokerPort 8000 -queryType sql -query select count(*) from baseballStats
limit 1 java.net.NoRouteToHostException: No route to host (Host
unreachable)```  
 **@mayanks:** Is broker up and running at port 8000 of 0.0.26.108?  

###  _#getting-started_

  
 **@datanarch01:** @datanarch01 has joined the channel  
 **@ciro.zar:** @ciro.zar has joined the channel  

###  _#releases_

  
 **@siddhartha.varma:** @siddhartha.varma has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org