Apache Pinot Daily Email Digest (2021-09-14)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@zineb.raiiss:** Hello friends, This is my first time on this platform. I
have a problem with a table on Pinot. I created Schéma and the table on Pinot
but I can't seem to populate the table with the data I have on the csv file. I
read the message I got during segmentation and found the date column field on
Schema is not declare well, Date field format on csv file does not match what
is declare on diagram. The Date field on File: 08/01/2018 08:00:00 Statement
on Diagram: "dateTimeFieldSpecs": [{ "name": "Date_measure", "dataType":
"STRING", "format": "1: DAYS: SIMPLE_DATE_FORMAT: dd / MM / YYY HH: mm",
"granularity": "5: MINUTES"  
**@kharekartik:** Can you try with format
"format":"1:DAYS:SIMPLE_DATE_FORMAT:dd/MM/yyyy HH:mm:ss"  
**@xiangfu0:** i think you have extra spaces in `dd / MM / YYY HH: mm`  
**@zineb.raiiss:** **, I don't have ss in my Data  
**@zineb.raiiss:** **, I add the space even if I have no space in the column?
this is the format I have on the CSV file: 01/08/2018 08:00 01/08/2018 08:05
01/08/2018 08:10  
**@zineb.raiiss:** This is result of Segment  
**@kharekartik:** Hi, I think just removing the ss from the format I suggested
should work then  
**@zineb.raiiss:** Hi ,  
**@zineb.raiiss:** yes, I have deleted ss since I have no second in the data.
But still, walk no longer  
**@zineb.raiiss:** here is format I used in
declaration"format":"1:DAYS:SIMPLE_DATE_FORMAT:dd/MM/yyyy HH:mm"  
 **@zineb.raiiss:** if you have any idea about the fault I put  
 **@xiangfu0:** Dear Community, TL;DR, Pinot will remove  after 0.9.0 release,
only  is supported afterward. Apache Pinot has been supporting the Calcite SQL
query endpoint from 0.3.0 (Mar 2020) and deprecating the PQL query endpoint
for more than a year. Now we are seeing more obstacles to maintain two code
paths for PQL & SQL for adding new functionalities, query optimization,
behavior handling, etc. So we decide to remove the PQL query path and
endpoint. If you are still on the PQL query path, please migrate to SQL
endpoint:  Current timeline is driven by the next Apache Pinot release: 1\.
Apache Pinot 0.9.0 release (ETA Release date: Oct/Nov 2021) is the last
official Pinot release that supports PQL. 2\. Apache Pinot master branch will
remove PQL support once the 0.9.0 release is cut, and no PQL support from the
0.10.0 release.  
 **@zineb.raiiss:** Hi,  
 **@zineb.raiiss:** after the execution of the segment in order to fill the
table here is the error message that I have  
 **@syed.hadi:** @syed.hadi has joined the channel  
 **@prtk.ngm:** Hi Team, We need to some preprocessing of the record before
running ingestion job - create pinot segments outside and push to Pinot data
store, what is the way in Pinot we can run some spark job (preprocessing)
before running actual ingestion job for pinot segments.  
**@mayanks:** What’s the preprocessing you need to do? Currently the spark
ingestion job in Pinot does not support custom preprocessing  
**@prtk.ngm:** we already have on preprocessing spark job - This spark job o/p
goes to HDFS  
**@prtk.ngm:** We have ingestion job read HDFS create segment and push to
pinot data store.  
**@prtk.ngm:** we want to create pipeline process data after that create
segment and push to pinot  
**@mayanks:** If you already have input for create segment you can just use
ingestion job to create segment and push to Pinot right?  
**@mayanks:** @prtk.ngm  
**@lxy1995seu:** @lxy1995seu has joined the channel  

###  _#random_

  
 **@syed.hadi:** @syed.hadi has joined the channel  
 **@lxy1995seu:** @lxy1995seu has joined the channel  

###  _#feat-better-schema-evolution_

  
 **@mattk:** @mattk has joined the channel  

###  _#pinot-website_

  
 **@mattk:** @mattk has joined the channel  

###  _#troubleshooting_

  
 **@syed.hadi:** @syed.hadi has joined the channel  
 **@lxy1995seu:** @lxy1995seu has joined the channel  
 **@bowenwan:** Hi, it seems that `bin/pinot-ingestion-job.sh` does not accept
yaml ingestion job config. It does not work with the example command
(`bin/pinot-ingestion-job.sh
examples/batch/airlineStats/ingestionJobSpec.yaml`) I copy from . It will
shows following error `Error: No argument is allowed:
examples/batch/airlineStats/ingestionJobSpec.yaml` . Can someone help me
what's wrong here ?  

###  _#feat-geo-spatial-index_

  
 **@mattk:** @mattk has joined the channel  

###  _#pinot-dev_

  
 **@zineb.raiiss:** @zineb.raiiss has joined the channel  
 **@mattk:** @mattk has joined the channel  
 **@suresh.intuit:** @suresh.intuit has joined the channel  
 **@weixiang.sun:** @weixiang.sun has joined the channel  

###  _#s3-multiple-buckets_

  
 **@mattk:** @mattk has joined the channel  

###  _#pinot-realtime-table-rebalance_

  
 **@mattk:** @mattk has joined the channel  

###  _#getting-started_

  
 **@zineb.raiiss:** @zineb.raiiss has joined the channel  
 **@zineb.raiiss:** Hello, This is my first time on this platform. I have a
problem with a table on Pinot. I created Schéma and the table on Pinot but I
can't seem to populate the table with the data I have on the CSV file. I read
the message I got during segmentation and found the date column field on
Schema is not declare well, Date field format on CSV file does not match what
is declare on the diagram. The Date field on File: 08/01/2018 08:00:00
Statement on Diagram: "dateTimeFieldSpecs": [{ "name": "Date_measure",
"dataType": "STRING", "format": "1: DAYS: SIMPLE_DATE_FORMAT: dd / MM / YYY
HH: mm", "granularity": "5: MINUTES"  
**@jackie.jxt:** Can you please try `dd/MM/yyyy HH:mm` and see if it works?  
**@jackie.jxt:** I do recommend transforming the time format when ingesting
the data using ingestion transform:  
**@jackie.jxt:** `/` is an invalid character in segment name fyi  
**@zineb.raiiss:** I corrected the Format- but doesn't work  
**@zineb.raiiss:**  
 **@zineb.raiiss:** Please help  
 **@suresh.intuit:** @suresh.intuit has joined the channel  
 **@dadelcas:** Hey, is there a way to tag servers upon deployment/start up?
I.e. via config files inatead of using the API?  
**@g.kishore:** No, you can only tag them using API.. we avoid config files as
much as possible.. it’s makes everything static  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org