[GitHub] ofajardo commented on issue #6446: Out of bounds nanosecond timestamp

[GitBox <gi...@apache.org>]

ofajardo commented on issue #6446: Out of bounds nanosecond timestamp
URL: https://github.com/apache/incubator-superset/issues/6446#issuecomment-459266621
 
 
   Same problem here. I have in my databases dates such as 9999-12-31 (yes, it is on purpose, no it is not an error, no, it cannot be changed =) ) 
   
   the problem can be easily reproduced as you said in pandas:
   
   ```python
   >>> pandas.to_datetime('9999-12-31')
   Traceback (most recent call last):
     File "/data/RWD_SHARE/MDH/APPS/superset_dev/supersetenv/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 377, in _convert_listlike
       values, tz = conversion.datetime_to_datetime64(arg)
     File "pandas/_libs/tslibs/conversion.pyx", line 188, in pandas._libs.tslibs.conversion.datetime_to_datetime64
   TypeError: Unrecognized value type: <class 'str'>
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/data/RWD_SHARE/MDH/APPS/superset_dev/supersetenv/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 469, in to_datetime
       result = _convert_listlike(np.array([arg]), box, format)[0]
     File "/data/RWD_SHARE/MDH/APPS/superset_dev/supersetenv/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 380, in _convert_listlike
       raise e
     File "/data/RWD_SHARE/MDH/APPS/superset_dev/supersetenv/lib/python3.6/site-packages/pandas/core/tools/datetimes.py", line 368, in _convert_listlike
       require_iso8601=require_iso8601
     File "pandas/_libs/tslib.pyx", line 492, in pandas._libs.tslib.array_to_datetime
     File "pandas/_libs/tslib.pyx", line 699, in pandas._libs.tslib.array_to_datetime
     File "pandas/_libs/tslib.pyx", line 671, in pandas._libs.tslib.array_to_datetime
     File "pandas/_libs/tslib.pyx", line 657, in pandas._libs.tslib.array_to_datetime
     File "pandas/_libs/tslibs/np_datetime.pyx", line 121, in pandas._libs.tslibs.np_datetime.check_dts_bounds
   pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 9999-12-31 00:00:00
   ```
   
   The thing is obviously that you cannot convert such a date to datetime64[ns] as that one is defined in nanoseconds. It should remain as datetime.datetime or string or whatever it was originally.
   
   I did a couple of quick changes on superset dataframe.py that solve it, but I don't know what is the consequence or if there is a better alternative.
   
   I changed: 
   
   ```
   pd.DataFrame(list(data), columns=self.column_names).infer_objects())
   ```
   
   to
   
   ```
   pd.DataFrame(list(data), columns=self.column_names)) # .infer_objects())
   ```
   
   because that infer_objects is trying to convert to datetime64ns.
   What is the consequence of doing this? Everything looks good in sqlab
   
   and then I had to do yet another change:
   
   ```python
   data = [dict((k, _maybe_box_datetimelike(v))
                   for k, v in zip(self.df.columns, np.atleast_1d(row)))
                   for row in self.df.values]
   ``` 
   
   to
   
   ```
   data = [dict((k, v)
                   for k, v in zip(self.df.columns, np.atleast_1d(row)))
                   for row in self.df.values]
   ```
   
   because that _maybe_box_datetimelike (by the way in pandas 0.24 the underscore is removed, I am working with pandas 0.23 for now) merciless transforms to datetime64ns. 
   
   I think I don't have datetimes with timezone information in my database so I hope it will be good for now ... or are these changes going to bite me very soon?
   
   suggestions appreciated.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org