Apache Pinot Daily Email Digest (2021-10-15)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@zjureel:** @zjureel has joined the channel  
 **@mmadou:** @mmadou has joined the channel  
 **@mustafaf:** @mustafaf has joined the channel  
 **@helario:** @helario has joined the channel  

###  _#random_

  
 **@zjureel:** @zjureel has joined the channel  
 **@zjureel:** @zjureel has left the channel  
 **@mmadou:** @mmadou has joined the channel  
 **@mustafaf:** @mustafaf has joined the channel  
 **@helario:** @helario has joined the channel  

###  _#troubleshooting_

  
 **@zjureel:** @zjureel has joined the channel  
 **@mmadou:** @mmadou has joined the channel  
 **@arawat:** ]  
 **@deemish2:** Hi All, If there is any gap in overlap data between realtime
and offline time ( e.g. batch ingestion job crashed in offline table ) . In
that case , is data available in hybrid table or not ?  
**@g.kishore:** To handle such cases Pinot does max(time) - 1 day from
offline. So as long as you rerun your batch ingestion job before the next days
run, it will be fine  
**@mustafaf:** @mustafaf has joined the channel  
 **@tony:** I am trying to configure the S3 storage to use server side
encryption and have
`pinot.controller.storage.factory.s3.serverSideEncryption=aws:kms`
`pinot.controller.storage.factory.s3.ssekmsKeyId=KEY` and I get the rather odd
error message ```Unknown value 'aws:kms' for S3PinotFS config:
'serverSideEncryption'. Supported values are: [AES256, aws:kms]```  
 **@helario:** @helario has joined the channel  
 **@kchavda:** Anyone using Pinot 0.8.0 and Superset? I upgraded Pinot to
utilize the TIMESTAMP datatype but now Superset is giving me error when trying
to create a dataset. Below is the error: ```Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: ERROR:root:'timestamp' Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: Traceback (most recent call last): Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-
packages/flask_appbuilder/api/__init__.py", line 84, in wraps Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: return f(self, *args, **kwargs) Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/superset/superset/views/base_api.py", line 80, in wraps Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: duration, response =
time_function(f, self, *args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File "/app/superset/superset/superset/utils/core.py", line
1368, in time_function Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
response = func(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File "/app/superset/superset/superset/utils/log.py", line
224, in wrapper Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: value =
f(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/superset/superset/datasets/api.py", line 236, in post Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: new_model =
CreateDatasetCommand(g.user, item).run() Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File
"/app/superset/superset/superset/datasets/commands/create.py", line 47, in run
Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: self.validate() Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/superset/superset/datasets/commands/create.py", line 87, in
validate Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: if database and not
DatasetDAO.validate_table_exists( Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File "/app/superset/superset/superset/datasets/dao.py",
line 81, in validate_table_exists Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: database.get_table(table_name, schema=schema) Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/superset/superset/models/core.py", line 603, in get_table Oct
15 17:11:23 ip-10-0-7-125 superset[1849861]: return Table( Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: File "<string>", line 2, in __new__ Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py",
line 139, in warned Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return
fn(*args, **kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line
560, in __new__ Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
metadata._remove_table(name, schema) Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File "/app/superset/lib/python3.8/site-
packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__ Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: compat.raise_( Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File "/app/superset/lib/python3.8/site-
packages/sqlalchemy/util/compat.py", line 182, in raise_ Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: raise exception Oct 15 17:11:23 ip-10-0-7-125
superset[1849861]: File "/app/superset/lib/python3.8/site-
packages/sqlalchemy/sql/schema.py", line 555, in __new__ Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: table._init(name, metadata, *args, **kw) Oct
15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line
644, in _init Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: self._autoload(
Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line
667, in _autoload Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
autoload_with.run_callable( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
File "/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/base.py",
line 2212, in run_callable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
return conn.run_callable(callable_, *args, **kwargs) Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-
packages/sqlalchemy/engine/base.py", line 1653, in run_callable Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: return callable_(self, *args,
**kwargs) Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line
469, in reflecttable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return
insp.reflecttable( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py",
line 664, in reflecttable Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: for
col_d in self.get_columns( Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
File "/app/superset/lib/python3.8/site-
packages/sqlalchemy/engine/reflection.py", line 390, in get_columns Oct 15
17:11:23 ip-10-0-7-125 superset[1849861]: col_defs = self.dialect.get_columns(
Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-
packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 390, in
get_columns Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: columns = [ Oct
15 17:11:23 ip-10-0-7-125 superset[1849861]: File
"/app/superset/lib/python3.8/site-
packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 393, in
<listcomp> Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: "type":
get_type(spec["dataType"], spec.get("fieldSize")), Oct 15 17:11:23
ip-10-0-7-125 superset[1849861]: File "/app/superset/lib/python3.8/site-
packages/pinotdb-0.3.6-py3.8.egg/pinotdb/sqlalchemy.py", line 458, in get_type
Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]: return
type_map[data_type.lower()] Oct 15 17:11:23 ip-10-0-7-125 superset[1849861]:
KeyError: 'timestamp'``` I've upgraded to latest pinotdb driver.  
**@richard892:** Looks like a bug, can you submit a github bug report?  
**@kchavda:** Hi @richard892, should I submit pinot or the python-pinot-dbapi
repo?  
**@richard892:** it looks like the python library needs updating to know about
timestamps  
**@kchavda:** Yeah, looks like the type_map doesn't have dates?  
**@richard892:** submit the feature request to the python library  
**@kchavda:** Just submitted issue.  
**@richard892:** If nobody gets round to it I can submit a patch next week  

###  _#pinot-dev_

  
 **@zjureel:** @zjureel has joined the channel  

###  _#getting-started_

  
 **@helario:** @helario has joined the channel  
 **@benshahbaz:** @benshahbaz has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org