[GitHub] betodealmeida commented on a change in pull request #4724: Improve database type inference

[GitBox <gi...@apache.org>]

betodealmeida commented on a change in pull request #4724: Improve database type inference
URL: https://github.com/apache/incubator-superset/pull/4724#discussion_r178584591
 
 

 ##########
 File path: superset/dataframe.py
 ##########
 @@ -41,26 +63,50 @@ class SupersetDataFrame(object):
         'V': None,   # raw data (void)
     }
 
-    def __init__(self, df):
-        self.__df = df.where((pd.notnull(df)), None)
+    def __init__(self, data, cursor_description, db_engine_spec):
+        column_names = []
+        if cursor_description:
+            column_names = [col[0] for col in cursor_description]
+        self.column_names = dedup(column_names)
+
+        # check whether the result set has any nested dict columns
+        if data:
+            has_dict_col = any([isinstance(c, dict) for c in data[0]])
+            df_data = list(data) if has_dict_col else np.array(data, dtype=object)
 
 Review comment:
   I tested the behavior with `[(1, 2, {'a': 'b'})]` as data, trying to understand the code. If we build the dataframe using `np.array` on data **with** a dict we get the following types for the columns:
   
   ```python
   >>> data = [(1, 2, {'a': 'b'})]
   >>> df = pd.DataFrame(np.array(data, dtype=object))
   >>> print(df.dtypes)
   0    object
   1    object
   2    object
   dtype: object
   ```
   
   Here Pandas doesn't know that the first two columns are integers. OTOH, using `list(data)`:
   
   ```python
   >>> data = [(1, 2, {'a': 'b'})]
   >>> df = pd.DataFrame(list(data))
   >>> print(df.dtypes)
   0     int64
   1     int64
   2    object
   dtype: object
   ```
   
   Now we have correct type information for all columns. So it seems to me that the `list(data) if has_dict_col else np.array(data, dtype=object)` is trying to preserve the type information.
   
   We can get the same result using `infer_objects()`:
   
   ```python
   >>> data = [(1, 2, {'a': 'b'})]
   >>> df = pd.DataFrame(np.array(data, dtype=object)).infer_objects()
   >>> print(df.dtypes)
   0     int64
   1     int64
   2    object
   dtype: object
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services